Systems and methods for monitoring objects at sporting events

ABSTRACT

A system for monitoring objects at sporting events or other types of events uses a wearable drone that has at least one camera or other sensor for capturing or otherwise sensing data. When the drone is to be used for monitoring, such as monitoring an object at a sporting event, the wearable drone may be detached from its user, and it may hover or otherwise fly within a certain position of an object to be monitored. While flying, the drone&#39;s sensor may be used to capture information, such as performance data or images, of the object during the sporting event.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 15/438,289, entitled, “Systems and Methods forMonitoring Objects at Sporting Events” and filed on Feb. 21, 2017, whichis incorporated herein by reference. U.S. patent application Ser. No.15/438,289 claims priority to U.S. Provisional Application No.62/297,528, entitled “Systems and Methods for Monitoring Objects atSporting Events” and filed on Feb. 19, 2016, which is incorporatedherein by reference. U.S. patent application Ser. No. 15/438,289 is acontinuation-in-part of and claims priority to U.S. patent applicationSer. No. 14/874,555, entitled “Systems and Methods for MonitoringObjects in Athletic Playing Spaces” and filed on Oct. 5, 2015, which isincorporated herein by reference.

RELATED ART

In general, it can be desirable to monitor an athlete or other objectsat sporting events in order to provide an assessment of the athlete'sperformance or other information indicative of the sporting events. Asan example, systems have been developed that help to train an athlete toperform better or more consistently by measuring a parameter indicativeof the athlete's performance and providing feedback indicative of themeasured parameter so that the athlete can be informed of how well he orshe performed during the sporting event. Additionally, some systems areused to monitor a sporting event in order to provide statistics or otherdata about the sporting event for entertainment or training purposes. Asan example, a system may monitor and report the length of a field goalin football, the speed of a baseball thrown by a pitcher, the speed ofan athlete while sprinting, or information indicative of a trajectory ofan object, such as a football, baseball, basketball, golf ball, hockeypuck, soccer ball, or volleyball.

The systems and methods used to monitor athletes or other objects atsporting events can be complex and expensive, requiring various types ofsensors. In addition, sensors are often mounted or installed atpredefined locations limiting the amount and/or type of data that can becaptured by the sensors. Efficient and inexpensive techniques formonitoring objects at sporting events and other types of events aregenerally desired.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure can be better understood with reference to the followingdrawings. The elements of the drawings are not necessarily to scalerelative to each other, emphasis instead being placed upon clearlyillustrating the principles of the disclosure. Furthermore, likereference numerals designate corresponding parts throughout the severalviews.

FIG. 1 is a block diagram illustrating an exemplary system formonitoring athletes or other objects at sporting events.

FIG. 2 is a block diagram illustrating an exemplary drone, such as isdepicted by FIG. 1.

FIG. 3 is a three-dimensional perspective of a wearable drone that maybe programmed for use in the monitoring system depicted by FIG. 1.

FIG. 4 is a three-dimensional perspective of the drone depicted by FIG.3 being worn around a wrist of a user.

FIG. 5 is a perspective view of a football player with a dronepositioned behind the football player.

FIG. 6 is a three-dimensional perspective of a basketball player with adrone positioned behind the basketball player.

FIG. 7A is a three-dimensional perspective of a golfer attempting a putton a green of a golf course.

FIG. 7B depicts an image of the green of FIG. 7A displayed to a user fordepicting an optimal trajectory for a putt of a golf ball on the green.

FIG. 7C depicts an image of the green of FIG. 7A displayed to a user fordepicting multiple trajectories for a putt of a golf ball on the green.

FIG. 8 is a three-dimensional perspective of the football player of FIG.5 while the football player is wearing at least one multi-lens camera.

FIG. 9 is a three-dimensional perspective of the basketball player ofFIG. 6 while the basketball player is wearing at least one multi-lenscamera.

FIG. 10 is a block diagram illustrating an exemplary processing system,such as is depicted by FIG. 1.

FIG. 11 is a block diagram illustrating an exemplary system formonitoring objects in athletic playing spaces.

FIG. 12 is a block diagram illustrating an exemplary processing system,such as is depicted by FIG. 11.

FIG. 13 shows an exemplary sensing system, such as is depicted by FIG.11, mounted on a pole above a basketball goal.

FIG. 14 depicts an exemplary depth map image captured by a depth sensor,such as is depicted by FIG. 11.

FIG. 15 depicts an exemplary depth map image captured by a depth sensor,such as is depicted by FIG. 11, after depth pixels corresponding to aplaying surface plane have been removed.

FIG. 16 depicts the exemplary depth map image depicted by FIG. 15 aftera hoop template has been superimposed on an image of a hoop.

FIG. 17 depicts an exemplary process for calibrating a gravity-basedcoordinate system.

FIG. 18 depicts a sensing system, such as is depicted by FIG. 11,coupled to an aerial vehicle, such as a drone or other aircraft.

DETAILED DESCRIPTION

The present disclosure generally pertains to systems and methods formonitoring objects at sporting events or other types of events. A systemin accordance with one embodiment of the present disclosure uses awearable drone that has at least one camera or other sensor forcapturing or otherwise sensing data. When the drone is to be used formonitoring, such as monitoring an object at a sporting event, thewearable drone may be detached from its user, and it may hover orotherwise fly within a certain position of an object to be monitored.While flying, the drone's sensor may be used to capture information,such as performance data or images, of the object during the sportingevent.

FIG. 1 depicts an exemplary system 10 for monitoring objects at sportingevents or other types of events. As shown by FIG. 1, the system 10comprises a movable drone 15 that is capable of flying to a desiredlocation in order to monitor objects. In one embodiment, the drone 15 isworn by a user, such as an athlete who intends to participate in asporting event, such as an athletic game or training session.

In the embodiment shown by FIG. 1, the drone 15 is detachably coupled toa user. As an example, the drone 15 may be mounted on or otherwisecoupled to a holding apparatus 17 that holds the drone 15 and isattachable to a user. In this regard, the holding apparatus 17 maycomprise a wristband that is wearable around the wrist of a user, andthe drone 17 may be detachably coupled to the wristband so that thedrone 17 may be detached from the wristband when it is to be used formonitoring. In an alternative embodiment, the drone 15 may form awristband that is detachably coupled to (e.g., wrapped around) the wristof a user, and the drone 15 may be detached from the user by unlockingthe wristband. In other embodiments, other types of bands (e.g.,armband, ankle band, headband, etc.) for temporarily securing the drone15 to other body parts are possible. In addition, the holding apparatus17 may comprise a clip or other type of coupling device for coupling theapparatus 17 to a clothing item or body part of the user. Various otherdevices and techniques for temporarily securing the drone 15 to a userare possible in other embodiments.

FIG. 2 depicts an exemplary embodiment of the drone 15. As shown by FIG.2, the drone 15 includes control logic 22 for generally controlling theoperation of the drone 15 as will be described in more detail hereafter.The control logic 22 can be implemented in software, hardware (e.g.,logic gates), firmware or any combination thereof. In the exemplarydrone 15 illustrated by FIG. 2, the control logic 22 is implemented insoftware and stored in memory 25.

Note that the control logic 22, when implemented in software, can bestored and transported on any computer-readable medium for use by or inconnection with an instruction execution apparatus that can fetch andexecute instructions. In the context of this document, a“computer-readable medium” can be any means that can contain or store acomputer program for use by or in connection with an instructionexecution apparatus.

The exemplary drone 15 depicted by FIG. 2 comprises at least oneconventional processor 28, such as a digital signal processor (DSP) or acentral processing unit (CPU), that communicates to and drives the otherelements within the drone 15 via a local interface 31, which can includeat least one bus. As an example, to implement any of the data processingfunctionality described herein for the drone 15, the processor 28 may beprogrammed with instructions of the control logic 22, when such controllogic 22 is implemented in software, for execution of the instructionsaccording to techniques known in the art.

As shown by FIG. 2, the drone 15 has an input interface 35 that may beused to receive inputs. As an example, the input interface 35 maycomprise a keypad, a keyboard, a mouse, buttons, switches, or othertypes of devices for receiving manual inputs or a microphone forreceiving audio inputs. The drone 15 also has an output interface 36that may be used to provide outputs. As an example, the output interface36 may comprise a display (e.g., a liquid crystal display) fordisplaying text, images, or other information or a speaker for providingaudio outputs. In some cases, a device (such as a touch-sensitivedisplay) may be used to implement the input interface 35 and the outputinterface 36 by both receiving inputs and providing outputs.

The drone 15 further comprises one or more sensors 44 for capturinginformation of interest during monitoring activities. As an example, asensor 44 may be implemented as an optical sensor (e.g., camera) forcapturing images of a scene. In one embodiment, a sensor 44 includes atwo dimensional camera for capturing two-dimensional images, and thedrone 15 also has a depth sensor 47 for sensing depth (e.g., a distancefrom the depth sensor 47 to one or more objects). As an example, thedepth sensor 47 may capture a depth image for use in monitoring objects,such as is described in more detail below and by U.S. patent applicationSer. No. 14/874,555, entitled “Systems and Methods for MonitoringObjects in Athletic Playing Spaces” and filed on Oct. 5, 2015, which isincorporated herein by reference. Using data from such a depth sensor47, a ground plane, such as an athletic playing surface, may be locatedin order to determine the direction of gravity. Such a depth sensor 47may operate using infrared radiation, as is done by the Kinect® camerasold by Microsoft Corporation, although other wavelengths are possible.Many conventional depth sensors, such as the Kinect® camera, generallyoperate by projecting a pattern (e.g., dots or lines) of infraredradiation (IR) across a scene and measuring the time it takes for the IRto be reflected back to the sensor. Changes in contour of the reflectingsurface affect the time required for energy to be reflected and/or theamount of light reflected (effectively “distorting” the pattern),thereby enabling the camera to estimate the contour based on thereturns. Use of such depth sensors in conditions susceptible to highamounts of ambient light, such as outdoors, can be problematic as theambient light, which is noise for the sensor measurements, canessentially “wash out” the IR returns making them undetectable. However,techniques have been developed to configure a depth sensor toselectively accept returns from the regions being scanned by theprojector in order to limit the amount of noise introduced by ambientlight, thereby enabling the depth sensor to be used in conditionsinvolving high amounts of ambient light.

In some embodiments, a sensor 44 may be implemented as a proximitysensor for sensing whether an object is within a certain proximity ordistance of the drone 15. Various other types of sensors may be used inother embodiments. Sensor data 49 indicative of the information sensedby the sensors 44 is stored in memory 25. Such sensor data 49 may be theraw data captured by the sensors 44 or may be processed data generatedby processing of such raw data by the control logic 22. In addition,when a sensor 44 is implemented as a camera, the sensor data 49 maydefine the images captured by the sensor 44. Note that it is unnecessaryfor the sensor 44 and the depth sensor 47 to be on the drone 15. As willbe described in more detail hereafter, the sensor 44 or the depth sensor47 may be worn by the user, such as on clothing or a head-mounteddisplay, or reside at other locations and wirelessly communicate withthe processing system 46.

The drone 15 also has a wireless communication interface 45 forpermitting it to communicate wirelessly with other devices. As anexample, the wireless communication interface 45 may comprise a radiofrequency (RF) radio capable of transmitting and receiving wireless RFsignals. As shown by FIG. 1, the drone 15 wirelessly communicates with aprocessing system 46 using the wireless communication interface 45 inorder to provide the sensor data 49 to the processing system 46. Ifdesired, the processing system 46 may be configured to transmit commandsfor controlling the operation of the drone 15. The processing system 46may also be configured to process and analyze the sensor data 49 as maybe desired. Note that the processing system 46 may be implemented inhardware or any combination of hardware, software, and/or firmware. Asan example, the processing system 46 may include one or more processorsprogrammed with instructions to perform the data processing functionsdescribed herein for the processing system 46. In this regard, theprocessing system 46 may be implemented as one or more computers, suchas a desktop, laptop, handheld (e.g., a smartphone), or mainframecomputer. In some embodiments, the processing system 46 may beintegrated with or otherwise reside on the drone 15 such that wirelesscommunication of the sensor data 49 is unnecessary, or the processingsystem 46 may be tethered to the drone 15 permitting the sensor data 49to be transmitted to the processing system 46 via a physical connection(e.g., one or more wires). In addition, it is possible for the drone 15to store sensor data 49 during monitoring and for the sensor data 49 tobe downloaded or otherwise provided to the processing system 46 aftermonitoring. The operation of the processing system 46 will be describedin more detail below.

In this regard, FIG. 10 depicts an exemplary embodiment of a processingsystem 46. As shown by FIG. 10, the processing system 46 includescontrol logic 122 for generally controlling the operation of the system46 as will be described in more detail hereafter. The control logic 122can be implemented in software, hardware (e.g., logic gates), firmwareor any combination thereof. In the exemplary processing system 46illustrated by FIG. 10, the control logic 122 is implemented in softwareand stored in memory 125. Note that the control logic 122, whenimplemented in software, can be stored and transported on anycomputer-readable medium for use by or in connection with an instructionexecution apparatus that can fetch and execute instructions.

The exemplary processing system 46 depicted by FIG. 10 comprises atleast one conventional processor 128, such as a digital signal processor(DSP) or a central processing unit (CPU), that communicates to anddrives the other elements within the system 46 via a local interface131, which can include at least one bus. As an example, to implement thefunctionality described herein for the processing system 46, theprocessor 128 may be programmed with instructions of the control logic122, when such control logic 122 is implemented in software, forexecution of the instructions according to techniques known in the art.

As shown by FIG. 10, the processing system 46 has an input interface 135that may be used to receive inputs. As an example, the input interface135 may comprise a keypad, a keyboard, a mouse, buttons, switches, orother types of devices for receiving manual inputs or a microphone forreceiving audio inputs. The processing system 46 also has an outputinterface 136 that may be used to provide outputs. As an example, theoutput interface 136 may comprise a display (e.g., a liquid crystaldisplay) for displaying text, images, or other information or a speakerfor providing audio outputs. In some cases, a device (such as atouch-sensitive display) may be used to implement the input interface135 and the output interface 136 by both receiving inputs and providingoutputs. The output interface 136 may be integrated with components ofthe processing system 46. As an example, the output interface 136 may bea display screen of a smartphone that has one or more processors forperforming the data processing functions described herein for theprocessing system 46.

The processing system 46 also has a wireless communication interface 145for permitting it to communicate wirelessly with other devices. As anexample, the wireless communication interface 145 may comprise a radiofrequency (RF) radio capable of transmitting and receiving wireless RFsignals. As shown by FIG. 1, the processing system 46 wirelesslycommunicates with the drone 15 using the wireless communicationinterface 145 in order to receive the sensor data 49. If desired, theprocessing system 46 may be configured to use the wireless communicationinterface 145 to transmit commands for controlling the operation of thedrone 15. The processing system 46 may also have a network interface147, such as a modem, for enabling the processing system 46 tocommunicate with a network (e.g., local area network (LAN), a wide areanetwork (VVAN), or other type of network). As an example, the processingsystem 46 may communicate with other devices via the Internet or othertype of network in order to provide access to the sensor data 49 orother information processed by the processing system 46. In addition, asnoted above, it is possible for components of the processing system 46to reside at various locations, including the drone 15. As an example,it is possible for the same processor to be used to execute instructionsof the control logic 22 shown by FIG. 2 and the control logic 122 shownby FIG. 10.

Note that the sensors 44 may include a location sensor, such as a globalpositioning system (GPS) sensor, for sensing the location (e.g.,geographical coordinates) of the drone 15. Such location sensor may beused to help navigate or position the drone 15 as may be desired. As anexample, the location of the drone 15 may be compared to a location ofanother object (such as a player) to move the drone 15 to a desiredlocation relative to the other object. As an example, a player ofinterest may wear a location sensor that transmits location coordinatesto the processing system 46, and the processing may 46 be configured tocompare such location coordinates to location coordinates received fromthe drone 15 in order determine a desired location for the drone 15(e.g., a predefined position from the player). The processing system 46then transmits commands to the drone 15 for moving it to the desiredlocation so that the drone 15 is a desired position from the player ofinterest. The location sensor may be used in other ways for otherembodiments.

The processing system 46 may be communicatively coupled to an outputinterface 50, such as a display device or a printer, for providingoutputs to users. In some embodiments, the output interface 50 isseparated from components of the processing system 46. In such examples,the output interface 50 may communicate with the processing system 46wirelessly or via one or more physical connections. As an example, theprocessing system 46 may be implemented as a laptop or other type ofcomputer that communicates wirelessly with a smartphone or other type ofcomputer having the output interface 50. Note that networks may be usedfor communication between components of the system 10. As an example, anetwork, such as a local area network (LAN) or wide area network (WAN),may be used for communication between the drone 15 and the processingsystem 46 or between the processing system 46 and the output interface50. In one embodiment, the processing system 46 is configured tocommunicate with one or more output interfaces 50 using the Internetand/or a cellular network, but other types of configurations arepossible in other embodiments.

As shown by FIG. 2, the drone 15 has a flight control system 52 forenabling the drone 15 to fly through air. As an example, the flightcontrol system 52 may have a controller and one or more propellers orother propulsion devices for propelling the drone 15 under the controlof the flight controller, as may be desired. Such flight controller maybe implemented in hardware or any combination of hardware, software,and/or firmware. For example, the flight controller may include one ormore processors programmed with software in order to implement the dataprocessing functions of the flight controller described herein.

The flight control system 52 may comprise any number and types ofairfoils (e.g., wings or rotors) for providing lift. As an example, FIG.3 shows a conventional drone 65 that can be programmed or otherwisecontrolled or modified to implement a drone 15 for monitoring athletes,as described herein. The drone 65 of FIG. 3 has a body 66 from whichfour flexible arms 69 extend. At the end of each arm 69 is a propeller73 that spins under the direction and control of a flight controller(not shown) housed within the body 66 in order to generate lift and topropel the drone 65 through air. The drone 65 also has a camera 77(e.g., a video camera or other type of camera) for capturing images asthe drone 65 flies. Note that the arms 69 are capable of bending anddetachably coupling to one another so that the drone 65 can be wornaround a user's wrist, similar to a watch, as shown by FIG. 4. Whenmonitoring is desired, the arms 69 can be detached from one another sothat the drone 65 can be removed from the user's wrist, and the arms 69may then be positioned as shown by FIG. 3 to permit the drone 65 to flyunder the control of its flight controller. Even though the arms 69 aresufficiently flexible to permit them to bend around a user's wrist, thearms 69 also have sufficient rigidity to permit them to maintain theirshape as the drone 65 is flying under the aerodynamic forces generatedby the propellers 73. In other embodiments, other types of drones may beused. As an example, a drone may have wheels, tracks, or other devicesfor enabling it to move along the ground or other surface.

There are various types of parameters that can be monitored by the drone15 during operation. As an example, the user wearing the drone 15 mayrelease it for flight just before engaging in a sporting activity suchas training for a sporting event or competing in a sporting event, suchas a game of basketball, football, baseball, golf, hockey, soccer,volleyball, skateboarding, and X games. After being released for flight,the drone 15 may be designed to hover or otherwise fly within a certainregion, such as a certain distance from the user or other object and/orat a certain height, and to capture sensor data 49 indicative of theuser's performance in the sporting activity. As an example, when theuser is taking a basketball shot by launching a basketball toward abasketball goal, at least one sensor 44 is configured to provide sensordata 49 indicative of the shot, and the processing system 46 isconfigured to analyze the sensor data 49 to determine one or moremetrics indicative of a quality of the basketball shot, such as releaseheight, release velocity, shot height, entry angle, entry velocity, shottrajectory, make/miss (i.e., whether the basketball passes through thehoop during the shot), or the ball speed or velocity at any point alongthe shot trajectory. Based on the sensor data 49, such as images of theuser taking a basketball shot, the processing system 49 may determineshot type, such as whether the shot is a jump shot, a lay-up, or athree-point shot. Note that the processing system 46 may use theshooter's location relative to the basketball goal at the time of theshot to determine shot type. For example, a shot with a certainproximity close to the goal on a side of the rim while the shooter ismoving horizontally may be determined to be a lay-up whereas anothershot greater than a certain distance from the goal may be determined tobe a three-point shot. Exemplary metrics that can be determined,analyzed, or otherwise processed by the processing system 46 aredescribed in U.S. Pat. No. 7,850,552, entitled “Trajectory Detection andFeedback System” and issued on Dec. 14, 2010, which is incorporatedherein by reference, and U.S. patent application Ser. No. 12/127,744,entitled “Stereoscopic Image Capture with Performance Outcome Predictionin Sporting Environments” and filed on May 27, 2008, which isincorporated herein by reference.

In some embodiments, the sensors 44 comprise a camera that is capable ofcapturing panoramic images (e.g., 360° views). Such a camera may beconfigured to capture multiple images as the camera is moved and then tostitch or otherwise combine the images together to form a panoramicimage. In another embodiment, the camera has multiple lenses thatreceive light from multiple directions thereby enabling the camera tosimultaneously capture images from different directions. The panoramiccamera is configured to stitch such images together to form a panoramicimage. As an example, the camera might have three lenses that areoriented at 120° relative to each other in order to capture a 360° viewaround the camera.

In some embodiments, the control logic 22 is configured to provideinputs to the flight control system 52 in order to automaticallyposition the drone 15 at a predefined position relative to a particularuser or other object. As an example, the control logic 22 may positionthe drone 15 a certain distance and/or height from an athleteparticipating in a sporting event. As the athlete moves, the controllogic 22 may sense his movements via use of the one or more sensors 44and then provide inputs to the flight control system 52 such that thedrone 15 is moved to track the movement of the athlete. As an example,the control logic 22 may attempt to keep the drone at a constantposition (e.g., distance and/or height) from the athlete as he moves.While the athlete is moving or otherwise participating in an event,information indicative of the his performance and movements is capturedby the sensors 44 and stored in memory 25 as sensor data 49.

The control logic 22 may be configured to distinguish the athlete ofinterest from other athletes at the event in order to assist the controllogic 22 in tracking the athlete's movements. As an example, at leastone of the sensors 44 may be configured to capture images of theathlete's face, and the control logic 22 may be configured to employknown facial recognition algorithms to distinguish the athlete ofinterest from other athletes. If the athlete of interest is wearing ajersey with a number printed on the jersey, the control logic 22 mayanalyze images of the user's jersey to distinguish him from otherathletes. As an example, the control logic 22 may analyze a capturedimage of the athlete to determine the jersey color and jersey number foridentifying the athlete. In this regard, in many sporting events, eachpossible jersey color and number combination is unique such that anyplayer can be identified by analyzing his jersey. In other embodiments,other techniques for identifying the athlete or other user of interestare possible.

In one example, the drone 15 is used to monitor a quarterback during afootball game or practice. The drone 15 may be positioned at a certainposition relative to the quarterback to permit suitable monitoring withvery little risk of the drone 15 being struck by the athletes or objectsin the game practice. As an example, the drone 15 may be positionedapproximately 20 feet in the air and approximately 20 feet behind thequarterback (i.e., on a side of the quarterback opposite of the line ofscrimmage). Thus, the drone 15 should remain at a sufficiently highaltitude such that it is not reachable by athletes during gameplay. Inaddition, when the quarterback throws a pass, he usually throws the passforward (toward and past) the line of scrimmage such that the positionof the drone 15 behind the quarterback reduces the likelihood of thedrone 20 being struck by the ball during a pass even though the drone 15may be at a height through which the ball may pass.

During gameplay, the drone 15 may be configured to capture images of thequarterback and/or other objects. As an example, the drone 15 maycapture a video feed and wirelessly transmit the video feed so that itcan be displayed for entertainment or other purposes. As an example, thevideo feed may be included in a television or other type of videobroadcast of the game. The video feed may also be used for otherpurposes. As an example, the video feed may be stored and laterdisplayed to the quarterback or coach for assisting with thequarterback's training.

In one embodiment, processing system 46 is configured to analyze theimages captured by the drone 15 and to identify the quarterback withinthe images. The processing system 46 also identifies the football withinthe images based on the football's color and shape. Based on the images,the processing system 46 is configured to determine when the football isthrown by the quarterback. There are various techniques that can be usedto determine when the football is thrown.

In this regard, the quarterback is generally expected to have a certainprofile, referred to hereafter as “throwing profile,” as he is throwingthe football where his forearm and hand are above his shoulder while hishand is gripping the football. While his forearm and hand are in suchprofile, the football is released by the quarterback such that thefootball separates from the quarterback's hand. The processing system 46may be configured to detect a throw when (1) the quarterback's forearmand hand are in the expected throwing profile and (2) the footballseparates from the quarterback's hand. Note that separation of thefootball from the quarterback when his forearm and hand are not in thethrowing profile may indicate the occurrence of another event, such as ahandoff, a pitch, or a fumble.

When a pass is detected, the processing system 46 is configured to trackthe trajectory of the pass and to calculate various trajectoryparameters indicative of pass quality. As an example, the processingsystem 46 may calculate release height, release angle, velocity,rotation rate, maximum pass height, pass distance, or other parametersthat may be of interest. Note that various parameters, such as releaseheight, pass height, and pass distance, may be determined through theuse of a depth sensor 47, which is capable of measuring the depth of theball (relative to the sensor) and comparing such depth to the depthsmeasured for a ground plane (e.g., athletic playing surface) or otherobjects within the depth image. Further, speed or velocity may becalculated by estimating a distance that the football travels betweentrajectory points and measuring the time required for the football totravel between the trajectory points. Dividing distance by time yieldsthe ball's speed.

In one embodiment, the location (e.g., coordinates in free space) of thefootball at various trajectory points are determined, and such pointsare used to estimate a trajectory curve representing the ball'strajectory during the pass. In this regard, once the ball is released,gravity is usually the predominant force that acts on the ball duringflight, and if the direction of gravity is known, various parameters,such as velocity at any point along the trajectory can be calculatedusing the estimated trajectory curve. In one embodiment, the processingsystem 46 determines the direction of gravity using techniques similarthose described in more detail below and by U.S. patent application Ser.No. 14/874,555, and converts the coordinates provided by the depthsensor 47 into a gravity-based coordinate system so that the directionof gravity relative to the ball's trajectory is known. Varioustrajectory parameters and techniques for determining trajectoryparameters are described by U.S. Pat. No. 7,850,552 and U.S. patentapplication Ser. No. 12/127,744.

It should be noted that various parameters may indicate the outcome ofthe pass and may be dependent on the actions or positions of anotherathlete, such as a receiver who is attempting to catch the pass. In thisregard, the images captured by the drone 15 may include the receiver, aswell as one or more defenders attempting to defend the pass. Theprocessing system 46 may identify the receiver who is attempting tocatch the pass, referred to hereafter as “target receiver,” usingvarious techniques. For example, by tracking the football, theprocessing system 46 may determine the trajectory location (referred tohereafter as “trajectory endpoint”) where the path or velocity of thefootball along its trajectory is materially disrupted, indicating thefootball has struck an object, such as the ground or an athlete of thefootball game or practice. The athlete wearing a certain color jerseyclosest to the trajectory endpoint may be identified as the targetreceiver.

By analyzing the images captured after the football reaches theendpoint, the processing system 46 can determine whether the footballwas caught (i.e., pass completed) by the target receiver. As an example,if the football appears to remain in the hands of the target receiverfor at least a threshold amount of time based on the video imagescaptured by the drone or other cameras, the processing system 46 maydetermine that the football was caught. If, on the other hand, the videoimages show that the football remains in the hands of a defender wearinga different color jersey for at least the threshold amount of time, thenthe processing system 46 may determine that the pass was intercepted. Ifthe football is determined to strike the ground before a pass completionor interception is made, then the processing system 46 may determinethat the pass was incomplete. In other embodiments, other techniques maybe used to determine the outcome status (e.g., completion, interception,or incompletion) of the pass. Note that sensors (e.g., cameras) inaddition or in lieu of the ones on the drone 15 may be used to provideinformation to the processing system 46.

Over time, the processing system 46 may collect and store variousstatistics regarding the quarterback's performance, such as totalattempts, total completions, total interceptions, completion percentage,interception percentage, average release height, average release angle,average velocity, average rotation rate, average maximum pass height,average pass distance, or other statistics that may be of interest.Further, in collecting statistics about multiple players, the processingsystem 46 is preferably configured to identify the athletes so that theappropriate data can be correlated with the identified athletes. As anexample, athletes may be identified based on the number on theirjerseys, as described above, or through other types of techniques, suchas facial recognition or other known techniques for identifyingindividuals.

Note that the performance data collected by the processing system 46 maybe categorized in any manner as may be desired. As an example, for aquarterback, statistics may be calculated based on pass distance. Forexample, total attempts, total completions, total interceptions,completion percentage, interception percentage, etc. may be calculatedfor passes thrown in a certain distance range, such as 0 to 10 yardswhile the same or similar statistics for another distance range, such as10 to 20 yards may be separately tracked. In addition, the processingsystem 46 may implement algorithms for calculating various qualitativeinformation about the passes.

As an example, based on the target receiver's location and velocity at acertain point, such as the quarterback's release of the football or at apoint while the football is in flight, the processing system 46 mayidentify a zone in the playing space, referred to hereafter as “targetzone” where the football is ideally thrown in order to complete a passto the target receiver. The target zone may also be based on variousother factors, such as the location and velocity of one or moredefenders at the time of the quarterback's release of the football or apoint while the football is in flight. The processing system 46 may alsocompare the trajectory of the pass to the target zone to determinewhether the pass is directed to the target zone (e.g., whether thetrajectory intersects with the target zone) or a distance that thetrajectory is from the target zone. Directing a pass to the target zoneor a small distance from the target zone generally may indicate a betterquality pass regardless of whether the pass is actually completed. Theprocessing system 46 is configured to calculate various parameters basedon the pass trajectory relative to the target zone. As an example, theprocessing system 46 may determine the trajectory's average distancefrom the target zone for a plurality of passes, noting that the targetzone may be at a different location for different passes. Further, thesize of the target zone may be based on a distance of the pass or otherfactors. For example, shorter passes may have smaller target zones andlonger passes may have larger target zones. The target zone data mayalso be categorized based on pass distance or other parameters. As anexample, the average distance from the target zone for passes within onedistance range (e.g., 0 to 20 yards) may be determined, and the averagedistance from the target for passes within another distance range (e.g.,20 to 40 yards) may be separately determined.

Note that the drone 15 may be configured to monitor the data 49 from thesensors 44 and to provide inputs to the flight control system 52 basedon such data 49. As an example, the control logic 22 may be configuredto identify the player being monitored (e.g., the quarterback in theinstant example) and to change the location of the drone 15 based onsuch player's movement. As an example, the control logic 22 may controlthe drone 15 such that it remains within a certain position or proximityof the quarterback as he moves. Thus, if the quarterback rolls left, thedrone 15 may automatically move left so that it remains directly behindthe quarterback. If the quarterback advances down the field, the drone15 may also move down the field to stay a certain distance behind thequarterback. In other embodiments, it is unnecessary for the drone 15 tobe within the field of play. As an example, the drone 15 may bepositioned in the air on the sideline (off of the field of play) andmove back and forth along the sideline based on movement of objects inthe field of play. For example, as a player of interest advances downthe field, the drone 15 may move in a corresponding manner along thesideline. The drone 15 also may be positioned at the end of the fieldsuch as behind the goalposts or end zone. The drone 15 may move along aboundary of other fields or courts of play for other sports, such assoccer, hockey, volleyball, basketball, tennis, etc.

The control logic 22 may also implement a collision avoidance algorithmin order to protect the drone 15 from damage and to prevent the drone 15from interrupting play of the game. As an example, based on at least onesensor 44, such as an image captured by a camera or a measurement by aproximity sensor, the control logic 22 may determine that collision withan object (e.g., football, goalpost, person, or other drone 15) isimminent and then provide inputs to the flight control system 52 in anattempt to move the drone 15 in a manner that avoids the collision. Thecontrol logic 22 also may be configured to take certain action, such aschange a state of the drone 15, in order to protect the drone 15 fromthe imminent collision. As an example, if a component (e.g., a camera orother sensor 44) of the drone 15 is extended, the control logic 22 mayretract such component in order to reduce the likelihood that thecomponent would be damaged by the collision. Various other actions maybe taken to protect the drone 15 from damage.

Also note that any of the athletes or other individuals (e.g., referees,coaches, trainers, cheerleaders, mascots, etc.) at the football game orpractice may be monitored by the drone 15 according to similartechniques described above for the quarterback, and athletes orindividuals at other sports may be similarly monitored as well. As anexample, the drone 15 may be configured to monitor the performance of afield goal kicker in a football game or practice. In such example, thedrone 15 may be positioned behind the kicker, on the sideline, behindthe goalposts, or at other locations as described above. Video images ofthe kicker may be captured while he is kicking the football. Based onthe images, the processing system 46 may determine various parametersindicative of kick performance or quality. As an example, the processingsystem 46 may measure foot speed during the kick or determine thelocation on the football where the kicker's foot strikes the football.In addition, as described above for a pass, the drone 15 may captureimages of the football while the football is in flight from a kick, andthe processing system 46 may analyze the captured images to determine atrajectory of the football. Based on the trajectory, the processingsystem 46 may determine various parameters, such as the football'svelocity, rotation rate, distance of travel, ascension angle (i.e., theangle, relative to horizontal or the playing surface, of the beginningof the trajectory while the football is rising after the kick) or otherparameters that may be of interest. Based on the trajectory and/orcomparison of images of the football relative to images of thegoalposts, the processing system 46 may also determine whether the fieldgoal was successful or where the football passes through or past thegoalposts. As an example, the processing system 46 may determine ahorizontal distance of the trajectory from a certain point, such as acenter of the make zone (i.e., a point halfway between the goalposts).If the field goal is unsuccessful, the processing system 46 maydetermine a horizontal distance of the trajectory from the nearestgoalpost (indicating how far the field goal was missed).

Note that the position of the drone 15 may be controlled based on asituational awareness of gameplay. As an example, the control logic 22may control the drone 15 to operate in one mode, referred to herein asthe “quarterback mode,” in which the drone 15 is operated to monitor theplay of the quarterback, as described above. When the offense attempts afield goal, the control logic 22 may transition the operation of thedrone into another mode, referred to herein as the “kick mode,” in whichthe drone 15 is operated to monitor the kicker's performance. As anexample, in the quarterback mode, the drone 15 may be positioned behindthe quarterback, as described above, and for the kick mode, the drone 15may be positioned at another location, such as on the sidelines tobetter capture certain parameters, such as ascension angle. Also, whenthe ball changes sides, such as after a turnover, the operation of thedrone 15 may be changed from a mode to monitor the quarterback on oneteam to a mode for monitoring the quarterback (or other player) of theother team.

There are various techniques that can be used to determine when totransition the operational mode of the drone 15. As an example, when anevent occurs for which there is to be a mode transition, a userobserving the game or practice may provide an input indicative of themode change. Such input may be received by an input interface of theprocessing system 46 or other device, such as a smartphone, computer(e.g., laptop), or other device capable of receiving inputs, and suchinput may be wirelessly transmitted to the drone 15. In response, thecontrol logic 22 may change the operational mode of the drone 15, asindicated by the input. In another embodiment, the drone 15 may beconfigured to receive the input directly from a user. As an example, oneof the athletes, such as the player being monitored, or other user maysignal an input through a particular body motion (such as waving hishand in a predefined manner) or by providing a voice command to thedrone. For a body motion, the processing system 46 may be configured toanalyze the images captured by the drone 15 to determine when theathlete or other user has signaled the input.

In other embodiments, the decision about when to change modes may bebased on data from the sensors 44. As an example, the processing system46 may analyze images captured by a camera of the drone 15 or otherdevice to determine which team is on offense and wirelessly transmitcontrol information to the drone 15 for causing the control logic 22 toposition the drone 15 according to the quarterback mode to monitor thequarterback of the offense. There are various techniques that can beused to determine which team is on offense. As an example, the drone 15or other device may capture an image of the scoreboard, which may beoperated to indicate which team is on offense (such as by displaying animage of a football next to the name or score of the team on offense).Based on the location of such image or other indication by thescoreboard, the processing system 46 may determine which team is onoffense.

It is possible for the processing system 46 to make certain situationaldecisions (such as which team is on offense) based on the activityoccurring in the field of play. As an example, before a play, it isoften the case that the teams huddle on their respective sides of thefootball. The defense often huddles closer to the football than theoffense. Thus, based on the location of a team's huddle relative to thefootball in the images captured by the drone 15, the processing system46 may determine whether that team is on offense. In another example,certain referees are often positioned on a particular side of the balldepending on which team is on offense. As an example, the head refereeis often positioned on the offense side of the ball. The processingsystem 46 may be configured to identify a referee (using useridentification techniques described above, such as facial recognition orclothing recognition) to determine which team is on offense based on thereferee's position relative to the football. In addition, as describedabove, the processing system 46 may be configured to recognize certainplayers, and it is often the case that a particular player, such as thequarterback, only plays on offense or defense. The processing system 46may be configured to determine which team is on offense based on whichplayer or players are on the field. As an example, if the processingsystem 46 determines that the quarterback of a particular team is on thefield of play, the processing system 46 may be configured to determinethat such team is on offense. In such case, the processing system 46 maytransmit a command to the drone 15 for causing the drone 15 to operatein a certain mode, such as the quarterback mode for monitoring thequarterback. When the quarterback of one team leaves the field of play,the processing system 46 may determine that the other team is on offensewhen the system 46 detects the presence of the other team's quarterbackon the field of play. If the quarterback of a team leaves the field ofplay and if the field kicker of that same team enters the field of play,the processing system 46 may determine that a field goal attempt isoccurring. In such case, the processing system 46 may transmit to thedrone 15 a command for causing the drone 15 to operate in the kick mode.In basketball, the processing system 46 may identify the player who isdribbling to determine which team is on offense. In other examples, thepresence of other types of players on the field or court of play may besensed in order to detect other types of game situations and to operatethe drone 15 in other types of modes.

In a basketball game or practice, the drone 15 may be positioned at anypoint above the court or along the boundaries of the court, as describedabove for a football field. In one example, the drone 15 is positionedin the air at a certain distance (e.g., about 20 feet) above a hoop of abasketball goal. Using the sensors 44, such as a video camera, and/or adepth sensor 47, the drone 15 may capture images of the basketball andthe players on the court. When a player attempts a shot on the goal, thedrone 15 may capture images of the shooter and the basketball as ittravels toward the hoop. The processing system 46 may be configured todetermine the ball's trajectory and various parameters indicative of theshooter's performance in the basketball shot. Exemplary techniques fordetermining such parameters are described in U.S. Pat. No. 7,850,552 andU.S. patent application Ser. No. 12/127,744.

The drone 15 may be configured to monitor the images captured by asensor 44 and to control movement of the drone based on such images. Asan example, the control logic 22 may be positioned a certain proximity(e.g., a certain distance and direction) from the shooter, such as about10 feet behind the shooter and about 10 feet in the air). As describedabove for the quarterback mode, the drone 15 may move in conjunctionwith the shooter's movements in order to maintain its relative positionwith the shooter.

Note that it is unnecessary for the control logic 22 within the drone 15to monitor the captured images in order to position the drone 15 asdescribed herein. As an example, for any of the embodiments describedherein, the processing system 46 may be configured to monitor thecaptured images and to control the movement of the drone 15 remotelybased on such images.

When the drone 15 is positioned behind the shooter or other player ofinterest, as described above, the images captured by the drone 15 have aperspective from the shooter's viewpoint. That is, the images closelyresemble what the shooter or other player being monitored sees duringgameplay. Such a feature may be beneficial for training or entertainmentpurposes. In this regard, the images may be recorded and later renderedto the shooter or other player of interest so that he can examinegameplay and his actions from essentially the same viewpoint that he wasseeing during gameplay. The images may also be broadcast or otherwiserendered to fans who can see gameplay from the viewpoint of the shooteror other player of interest. This type of viewpoint may be provided forother sports, as well. For example, positioning the drone 15 behind thequarterback in the quarterback mode described above allows users to seegameplay from the quarterback's approximate viewpoint. In someembodiments, the quarterback may wear a location sensor to provide tothe processing system 46 data indicative of the quarterback's location,and the processing system 46 may be configured to convert thecoordinates of the image data into coordinates that are relative to acoordinate system associated with the quarterback so that the viewpointof the images matches that of the quarterback, as described in moredetail below with respect to a putter in golf.

The drone 15 may be used to monitor golfers. In this regard, aparticular golfer may be identified in the images captured by the drone15 or by other type of sensor data 49 using any of the recognitiontechniques described herein, and the drone 15 may be positioned acertain proximity from the identified golfer, as described above inother sports and as shown by FIG. 7A. As an example, the drone 15 may bepositioned at a certain location relative to the golfer to permit thesensors 44 to capture the golfer's swinging motion as well as ballflight. As described above for other sports, the processing system 46may be configured to determine the ball's trajectory based on the imagesor other type of sensor data captured by the drone 15. Based on thegolfer's body or club motion during his swing and/or the trajectory ofthe golf ball, the processing system 46 may be configured to determinevarious parameters indicative of the golfer's performance.

When the golfer is putting, the processing system 46 may be configuredto analyze the images captured by the drone 15 in order to determine thetopography, including slope, of the putting surface (e.g., the green).Using techniques similar to those described below and in U.S. patentapplication Ser. No. 14/874,555, the processing system 46 may beconfigured to determine the direction of gravity within the images inorder to determine the slope of the green's surface relative to gravity.As an example, the processing system 46 may be configured to convert thecoordinates provided by a depth sensor 47 or other type of opticalsensor from the sensor's coordinate system to a gravity-based coordinatesystem. As described further in U.S. patent application Ser. No.14/874,555, the direction of gravity may be defined by first identifyinga large flat plane within the drone's viewing area from a depth sensor47. The processing system 46 may assume that the direction of gravity isat a predefined angle (e.g., 90 degrees) relative to the identifiedplane. In other embodiments, the drone 15 may have a plurality ofaccelerometers, and the direction of gravity may be determined usingreadings from the accelerometers according to known techniques. Theprocessing system 46 may also identify an object and determine that thedirection of gravity is at predefined angle relative to the object. Asan example, in golf, the processing system 46 may be configured toanalyze the images captured by a sensor 44 to identify the green hole atwhich the ball is to be putted. The processing system 36 may identifythe ring formed by the lip of the hole and determine that the directionof gravity is perpendicular to the plane defined by such ring. Yet othertechniques for determining the direction of gravity are possible inother embodiments.

In other embodiments, different techniques may be used to determine thetopology of the putting surface. As an example, data indicative of thetopology of the green (including hole placement) may be predefined andstored in a database or other form of memory accessible to theprocessing system 46.

In addition to determining the green's topology and the direction ofgravity, the processing system 46 may also be configured to identify thegreen's hole, as described above, and the golfer's ball within theimages. Based on the topography of the putting surface, the location ofthe hole relative to the golfer's ball, and the direction of gravity,the processing system 46 may be configured to calculate or otherwisedetermine an optimal path for the ball in order for the golfer to makethe putt (i.e., for the ball to be putted into the hole). The processingsystem 46 may then provide feedback information to the golfer indicativeof such path. As an example, the processing system 46 may use the outputinterface 50 or 136 to display an image of the green, including thelocation of the hole and the golfer's ball. Within such image, theprocessing system 46 may display a virtual curve extending from the ballto the hole along the path corresponding to the optimal path determinedby the processing system 46. Thus, by viewing the image and specificallythe virtual curve, the golfer is able to see the optimal path of theball for the putt.

Note that there are various ways that feedback can be provided to thegolfer. As an example, the image of the green described above may bedisplayed on the golfer's smartphone or other hand-held or mobile devicecarried by the golfer. As an example, FIG. 7B shows an exemplary image200 that may be displayed to the user on an output interface 50 of amobile device, such as a smartphone. The image 200 shows a green 205having a hole 207 with a flag 211 positioned in the hole 207 to mark theplacement of the hole 207. The image 200 also shows a golf ball 215resting on the green 205 and a virtual curve 218 representing theoptimal path determined by the processing system 46 for the user's putt.Note that the image 200 may be captured by the sensor 44 of a drone 15or otherwise, such as by a camera that may be mounted at a fixedlocation near the green 205.

In one embodiment, the virtual curve 218 is displayed in an augmentedreality environment. As an example, as shown by FIG. 7A, the golfer maywear an augmented-reality head-mounted display (HMD) 216, such asaugmented-reality eyeglasses 216, that permit light to pass through thelenses of the augmented-reality HMD 216 so that the golfer can see thephysical surface of the green 205 and other objects, such as the hole207 and flag 211. The augmented-reality HMD 216 may then generate animage of the virtual curve 218 corresponding to the optimal path suchthat the virtual curve 218 generated by the interface 50 appears to besuperimposed on the physical surface of the green 205 seen by thegolfer. Other techniques for providing feedback to the golfer about theoptimal path are possible in other embodiments.

In addition, it is possible for the processing system 46 to displaymultiple paths that can be selected by the user. In this regard, thepath of a successful putt depends not only on green topology but alsothe pace of the golf ball during the putt. In this regard, a golf ballputted with a greater force tends to have greater momentum which maychange the path that the ball must take to reach the hole whentraversing across a sloped surface. Thus, for any given putt, there aretypically multiple paths that would lead to a successful outcomedepending on pace. The processing system 46 may display multiple virtualcurves representing such paths and/or provide feedback indicative of thedesired pace for a particular path. As an example, one virtual curve 222may be color coded one color for a firm putt, and another virtual curve218 may be color coded a different color for a soft putt, as shown bythe image 220 depicted by FIG. 7C.

As described above, images captured by the sensor 44 of the drone 15 orby the sensor 44 residing at other locations may be displayed to theuser wearing augmented-reality HMD 216. In such a situation, it may bedesirable to change the viewpoint of the image so that it is relative tothe location of the HMD 216 instead of the location of the sensor 44. Toperform such conversion, the processing system 46 is preferably aware ofthe approximate location of the sensor 44 and the approximate locationof the HMD 216. Using these locations, the processing system 46 can beconfigured to adjust the images captured by the sensor 44 so that theyappear as if they have been captured by the HMD 216. In adjusting theimages, the processing system 46 may be configured to change theorientation of the images to account for differences in the viewingangle of the sensor 44 relative to the viewing angle of the user throughthe HMD 216. As an example, the coordinates of the image may beconverted into coordinates that are relative to the coordinate systemused by the HMD 216 so that the displayed image has the properperspective for viewing by the user wearing the HMD 216. In someembodiments, the sensor 44 may reside on the HMD 216 or be otherwisepositioned such that such a conversion is unnecessary.

Note that there are various techniques that can be used for theprocessing system 46 to determine the locations of the sensor 44 and theHMD 216. As an example, it is possible for the sensor 44 to be a fixedlocation, such as mounted near the green, and the location of the sensor44 may be stored in memory of the processing system 46. If the sensor 44is on a drone 15, as described above, the drone 15 may have a locationsensor, such as a global positioning system (GPS) sensor, for determinethe location of the drone 15, and data indicative of this location maybe transmitted to the processing system 46. In addition, the HMD 216 maybe similarly equipped with a location sensor, and data indicative of thelocation of the HMD 216 can be transmitted to the processing system 46.Note that it is possible for the processing system 46 to reside on theHMD 216 or otherwise be situated on the user such that wirelesscommunication of the HMD's location information is unnecessary. In otherembodiments, it is possible to use radio frequency (RF) devices at knownlocations (e.g., fixed positions on the golf course) to communicate withthe sensor 44, drone 15, and/or HMD 216 to determine their respectivelocations using triangulation or some other algorithm for determininglocations of objects.

In soccer, the drone 15 may be positioned at a certain location relativeto a particular player, such as the player in possession of the soccerball, the goalie, a player taking a penalty or corner kick, or otherplayer of interest, as described above for other sports. As an example,the drone 15 may be positioned a certain distance (e.g., about 10 feet)behind the player at a certain height (e.g., about 10 feet) above theplaying surface. Alternatively, the drone 15 may be positioned on theside of the soccer field (e.g., the drone 15 may move up and down thesidelines). As described above for other sports, the drone 15 may bemoved based on the captured images to keep the drone 15 at a certainposition relative to the player of interest. In one embodiment, theposition of the drone 15 is controlled based on the location of thesoccer ball. As an example, the drone 15 may be controlled to hover overthe ball at a certain height (e.g., about 10 feet) and move inconjunction with the ball so that the drone 15 continues to hover overthe ball or at another predefined position relative to the ball. Asdescribed above for other sports, the processing system 46 may beconfigured to collect ball and player data based on the captured images,to determine ball trajectory for various kicks, and to determine variousparameters indicative of the player performance.

In tennis, the drone 15 may be positioned at a certain location relativeto a particular player of interest, as described above for other sports.As an example, the drone 15 may be positioned a certain distance (e.g.,about 10 feet) behind the player at a certain height (e.g., about 10feet) above the playing surface. Alternatively, the drone 15 may bepositioned on the side of the tennis court (e.g., the drone 15 may moveup and down the sidelines). As described above for other sports, thedrone 15 may be moved based on the captured images to keep the drone 15at a certain position relative to the player of interest, and theprocessing system 46 may be configured to collect ball and player databased on the captured images, to determine ball trajectory, and todetermine various parameters indicative of the player performance.

If desired, multiple drones 15 may be used to monitor gameplay. As anexample, multiple drones 15 may be used to track different players atthe same time, according to the techniques described above. Furthermore,additional drones 15 and/or sensors may be positioned at variouslocations to provide additional viewpoints. The processing system 46 maybe configured to stitch together multiple images from multiple drones 15or sensors at different locations in order to provide a larger,composite image of gameplay.

When multiple drones 15 are used, the drones 15 may implement collisionavoidance algorithms, as described above, in an effort to avoid eachother. In this regard, one drone may have a sensor 44 (such as aproximity sensor) for detecting another drone 15, and control its flightpath based on such sensor in a manner to avoid the other drone 15. Inone embodiment, the drones 15 are configured to communicate with eachother wirelessly in order help avoid collisions. As an example, thedrones 15 may communicate location information to each other. Forexample, a first drone 15 may transmit its location coordinates to asecond drone 15, which uses such coordinates to determine the locationof the first drone 15 so that it can control its flight path to avoidthe first drone 15. In another example, when a collision between twodrones 15 is determined to be imminent, one of the drones 15 cantransmit information of its intended flight path to the other drone 15,which uses such information to select a flight path that avoids acollision. Thus, when a first drone 15 determines to take evasivemaneuvers to avoid a second drone 15, the second drone 15 may be awareof the flight path of the first drone 15 that will result from theevasive maneuvers thereby helping the two drones to avoid each other.

As described above, there are various techniques that may be used tocontrol the drone 15, and such control may be autonomous,semi-autonomous, or manual. In addition, decisions on which sensors 44to enable or use can be made at any time and can be autonomous,semi-autonomous, or manual. Sensor calibration and position finding mayalso be autonomous, semi-autonomous, or manual. The processing system 46may include one or more user interfaces for receiving user inputs thatare converted into commands wirelessly transmitted to the drone 15. Itis possible for a single user to provide such inputs or for the controlto be dispersed across a number of users. As an example, control may behanded off from one operator to another, and the operators may be at thesame premises or remote from each other. The inputs from the operatorsmay be used to control the flight of the drone 15 and/or otheroperational aspects, such as sensor attributes (e.g., the focal lengthof a camera.

When monitoring of an event is completed, the drone 15 may be returnedor moved to a predetermined location for storage or other purposes. Anoperator could guide the drone 15 to such location, or the drone 15 canbe configured to automatically fly or otherwise move the drone 15 to thelocation. As an example, coordinates of the predetermined location maybe pre-programmed into the control logic 22, which is configured toprovide inputs to the flight control system 52 such that the drone 15automatically flies to the predetermined location.

As an example, the drone 15 may be stored at a base. At the base, thedrone 15 may be connected to a power source, such as a battery or poweroutlet, in order re-charge one or more electrical power sources (e.g.,batteries) on board the drone 15. When a person begins to perform asporting activity of interest, the sporting activity may be sensed by asensor 44 of the drone 15 or otherwise, and the drone 15 in response mayautomatically leave its base and fly to a position for monitoring thesporting activity, as described above. Once the sporting activity isfinished or monitoring is otherwise no longer desired, the drone 15 mayautomatically fly back to its base or other location for storage.

As an example, assume that the drone 15 is to be used to monitorbasketball players while practicing. When a player arrives at abasketball court and begins to dribble or shoot on the basketball court,a sensor 44 of the drone 15 or other sensor (e.g., a sensor mounted at afixed location close to the court) may detect the sporting activity(e.g., dribbling or shooting), may detect the presence of the player ator near the court, or may receive an input from the player or other userindicating that monitoring is to commence. In response to any of theseevents, the drone 15 may automatically fly to a desired location formonitoring the sporting activity and begin monitoring as describedabove.

In this regard, the drone 15 may find certain references to assist withmonitoring of the sporting activity. As an example, the drone 15 mayidentify the player and then fly to a predefined location relative tothe player being monitoring. In another example, the drone 15 may find aportion of the basketball court, such as a hoop on the basketball goalor court markings, and fly to a predefined position relative to thehoop, other portion of the basketball goal, or court markings. Yet othertechniques for orienting and positioning the drone 15 in or near theplaying space are possible.

Once the player has stopped the sporting activity for at least apredefined amount of time, has left a certain proximity (e.g., left thecourt), or has otherwise indicated that monitoring is to stop (e.g.,provided a user input), the drone 15 may then automatically return tothe base or other location for storage until the next event formonitoring occurs. Similar techniques may be used for monitoring othertypes of activities for basketball or other sports.

Information collected by the drone 15, including images or other sensordata captured by the drone 44, may be provided to users as part of asubscription of a service. As an example, the information may bewirelessly transmitted from the drone 15 to the processing system 46,which stores the information in memory, such as at a database accessibleby a plurality of subscribers. Access to the information may becontrolled by a server that is in communication with a network, such asthe Internet. As part of the service, the subscribers may access theserver through the network and download the information from the server.As an example, a video stream of images captured by the drone 15 may bestreamed from the server to one or more users. Such video stream may bedelivered in real-time as the sporting event is occurring, or the videodata may be stored at the server for later (e.g., on-demand) access bysubscribers. Other types of information collected by the drone 15 maysimilarly be delivered in real-time or stored for later access bysubscribers. If desired, the collected information may be curated forsocial media use.

In addition, information collected by the drone 15 may be used as inputfor video games, including virtual reality video games and augmentedreality video games. For example, movements of the athletes and otherobjects (such as football players and the football in a football game)may be captured and recorded by the drone 15. Such information may thenbe used to recreate an animation of such movements as part of a videogame or other viewing by users. Using a head-mounted display (e.g.,virtual-reality or augmented-reality eyeglasses) or other type ofdisplay, images of the athletes and other objects in the sporting eventmay be displayed to the user to generate a virtual-reality oraugmented-reality environment in which the user may participate. As anexample, for a football game, a user may go to a football field and usea head-mounted display, which permits the user to see the actualfootball field while projecting images of the recorded athletes into theuser's eyes so that the user sees images of the athletes as if thesporting event is occurring on the football field where he is located.Alternatively, an image of a virtual football field may be projectedsuch that the user sees the images of the athletes as if they areplaying on the virtual football field. The system may be interactivesuch that, as the user takes action within the video game, gameplay isaffected. For example, if the user moves within the virtual or augmentedreality environment close to a ball carrier, the image of the ballcarrier might be updated to reflect that he is tackled. Similartechniques may be used to provide video games for other types of sports.

As indicated above, the sensors 44 may include a video camera forcapturing images of a scene, such as frames of video images that can bestreamed to one or more users. In some embodiments, the drone 15 mayhave at least one multi-lens camera for capturing panoramic images of awide viewing angle, such as greater than 180°. As used herein, a“multi-lens” camera refers to a camera that has multiple lenses forreceiving multiple images and that stitches together or otherwisecombines images from multiple lenses to form a composite image having aviewing angle greater than the images receive by any one of the lenses.It is possible to implement such a multi-lens camera using multiplecameras by communicatively connecting the cameras to a processing systemthat stitches the images from the cameras to form a composite image. Inat least one embodiment, the drone 15 has a multi-lens camera capable ofcapturing 360° panoramic views, sometimes referred to as a “360°camera,” but other types of cameras and images are possible. As anexample, 360° cameras have been developed that provide 360° panoramicview without stitching, and the drone 15 may use such a camera, ifdesired. Cameras having wide horizontal viewing angles, such as about180° degrees or greater, shall be referred to herein as “wide-angle”cameras regardless of whether stitching is used. Such wide-anglecameras, when worn by a user, may be used to capture images of objectson opposite sides of the user, such as objects in front of and behindthe user.

In some embodiments, the drone 15 also has at least one depth sensor 47that is configured to capture a panoramic depth image of the same orsimilar view relative to the wide-angle camera described above. Thispanoramic depth image may have a wide viewing angle, such as greaterthan 180°. In one embodiment, a two-dimensional (2D) video cameraprovides a panoramic image with a viewing angle up to 360° while thedepth sensor 47 provides a corresponding depth image that has the sameor similar viewing angle and can be used to determine the depths ofobjects that appear in the image from the 2D wide-angle camera. Asdescribed above for panoramic video images, images from multiple lensesof the depth sensor 47 may be stitched together to form a compositedepth image, but for some depth sensors 47, stitching may be unnecessaryto achieve a wide-angle view.

In some embodiments, the athletes of a sporting event being monitored bythe drone 15 wear at least one wide-angle camera, which providespanoramic video images from the viewpoint of the user wearing suchcamera. Image data from the camera may be wirelessly transmitted to thedrone 15 and/or the processing system 46. Like the image data from acamera on the drone 15, image data from the wide-angle camera on theuser may be stored and displayed by the processing system 46 for viewingby subscribers or other users. In other embodiments, other types ofcameras may be worn by the user, as may be desired. Note that such acamera may be worn by an athlete to provide video images from theperspective of the athlete without the use of the drone 15.

Various techniques may be used to attach a wide-angle camera to a user,such as an athlete in a sporting event. In one embodiment, thewide-angle camera has a plurality of cameras, referred to as “componentcameras,” each of which has a smaller viewing angle than the panoramicimage provided by the wide-angle camera. Each component camera capturesan image of a sector within such camera's view and sends the image to aprocessing system, which stitches together such images to form awide-angle panoramic image, such as a panoramic image having a 360°viewing angle. As an example, there may be three component cameras whereeach component camera captures an image having a viewing angle ofapproximately 120° in order to form a combined image having a 360°viewing angle. In other embodiments, other numbers of component cameraswith different viewing angles are possible.

In one embodiment, each component camera is mounted on a base that canbe attached to a user. As an example, the base may be a band (e.g., aring) that fits around a body part of a user to secure the base and,therefore, the components to the user. Such a band may be rigid orelastic. As an example, the band may be worn around the user's torso(e.g., chest, waist, or hips), arm, leg, or head. For football orbaseball, the band may fit around the helmet of a football or baseballplayer. If desired, the component cameras and/or band may be integratedwith (e.g., embedded within) the helmet or other headgear. Regardless ofthe techniques used to secure the component cameras to the user, thecomponent cameras may form a ring of cameras capable of capturing imagesfrom opposite sides of the user, such as images from the user's frontand back. For water polo, a ring of cameras may be positioned on theuser's swim cap. In other sports, a ring of cameras may be mounted onother types of headgear, such as caps (e.g., baseball), hats, helmets,headbands (e.g., soccer or basketball), etc. In other embodiments, aring of component cameras may be positioned around other body parts, asdescribed above.

If desired, multiple rings of component cameras may be used. As anexample, one ring of component cameras may be positioned around a user'shead, and another ring of component cameras may be positioned around theuser's torso. The panoramic image from one ring may be combined (e.g.,stitched) with the panoramic image from the other ring to form acomposite panoramic image. Alternatively, one ring of cameras maycapture 2D images, and a ring of depth sensors may capture depthinformation for the captured images.

As an example, FIG. 8 shows a football player having a multi-lens camera(e.g., a plurality of component cameras) embedded in a helmet 301 of thefootball player. The component cameras may be positioned on an interiorof the helmet 301 such that they are not visible in FIG. 8. Each of thecomponent cameras may have a lens 306 that passes through the helmet301, is positioned on an exterior surface of the helmet 301, ispositioned to receive light through a hole in the helmet 301, or isotherwise positioned on the helmet 301 to receive light from a scenethat includes other players at the sporting event. In the embodiment ofFIG. 8, the lenses 306 are arranged in a ring around the helmet 301. Awireless communication interface may also be coupled to the helmet 301or otherwise positioned on the player to receive the images capturedthrough the lenses 306 and to wirelessly transmit the images to theprocessing system 46, which may stitch or otherwise combine the imagestogether form a composite image, such as a 360 degree view of thescenery around the player.

In FIG. 8, the player also has a band 309 around his waist. Componentcameras may be embedded in the band 309 or otherwise coupled to the band309. Each such component camera has a lens 306 that passes through theband 309, is positioned on an exterior surface of the band 309, ispositioned to receive light through a hole in the band 309, or isotherwise positioned on the band 309 to receive light from a scene thatincludes other players at the sporting event. As with the componentcameras coupled to the helmet 301, the component cameras coupled to theband 309 may provide captured images to a wireless communicationinterface that transmits the images to the processing system 46, whichmay stitch the images together form a composite image, such as a 360degree view of the scenery around the player. In addition, theprocessing system 46 may stitch or otherwise combine the composite imagederived from the component cameras coupled to the helmet 301 with thecomposite mage derived from the component cameras coupled to the band309 to form a larger composite image, such as a larger 360 degree viewof the scenery around the player. Note that other techniques forenabling a player to wear a multi-lens camera are possible. As anexample, component cameras or lenses may be embedded or otherwiseincorporated into the player's jersey and positioned around the playerto provide a 360 degree view rather than using band 309, or the band 309itself may be embedded or otherwise incorporated into the player'sjersey. Yet other techniques are possible in other embodiments.

FIG. 9 shows an embodiment in which a basketball player is wearing aband 321 around her head, similar to the band 309 of FIG. 8. In thisregard, component cameras may be embedded in the band 321 or otherwisecoupled to the band 321. Each such component camera has a lens 306 thatthat is exposed to receive light from a scene that includes otherplayers at the sporting event. As with the component cameras coupled tothe helmet 301, the component cameras coupled to the band 321 mayprovide captured images to a wireless communication interface thattransmits the images to the processing system 46, which may stitch orotherwise combine the images together form a composite image, such as a360 degree view of the scenery around the player.

Note that it is unnecessary for the component cameras to be rigidlycoupled to one another or for one ring of component cameras to berigidly coupled to another ring of component cameras. That is, it ispossible for the orientation and/or location of one component camera tochange during monitoring relative to the orientation and/or location ofanother component camera. In one embodiment a ring of component camerasfor capturing 2D video images is used in conjunction with a ring ofdepth sensors 47 for capturing multiple depth images from theperspective of the athlete of the sporting activity. The depth imagesmay be stitched or otherwise combined together to form a composite depthimage that may be used to determine a distance of an object in anydirection from the athlete. Using the depth image from one or more depthsensors 45, the playing surface plane can be identified to assist indetermining the direction of gravity, as described in U.S. patentapplication Ser. No. 14/874,555.

In addition to capturing images from the perspective of an athlete in asporting activity, the drone 15 may be used to capture images of theathlete wearing the wide-angle camera. This may help to gain anunderstanding of the athlete's thought's and state of mind. As anexample, a camera on the drone 15 or a camera at another location may beused to capture images of the athlete's face, and such images can beanalyzed to ascertain clues to the athlete's state of mind. For example,the direction that the athlete is looking can be determined andrecorded. In addition, the athlete's facial expression can be analyzedin an effort to estimate the degree to which the athlete is panicked orfocused. As an example, smiling may indicate that the athlete isrelaxed, and rapid eye movement without a smile may indicate that theathlete is nervous or panicked.

The video images captured by the wide-angle camera may be used toprovide a virtual reality or augmented reality environment to a user.From such video images, the user may be able to view the sporting eventfrom the perspective of the athlete that is wearing the wide-anglecamera. As an example, when the wide-angle camera is worn by aquarterback in a football game, the user viewing the video feed may beable to see, from the perspective of the quarterback, defensive linemanrushing the quarterback and receivers running routes. Using multiplerings of component cameras may help to increase the vertical viewingangle for the captured images. Also, using a wide-angle camera, it ispossible to see lineman approaching the quarterback from manydirections, such as in front of and behind (the quarterback's blindside)the quarterback. In other embodiments, the multiple rings may be usedfor other purposes. For example, as described above, at least one cameraon one ring may provide two-dimensional video images while at least onedepth sensor 47 on another ring may be used to provide depth informationfor the pixels captured by the camera.

Note that on a moving object, such as a drone 15, it is possible for theimages from a camera to be used to determine depth without using a depthsensor 47. In this regard, if a stationary object (e.g., a line on afield or court, a basketball hoop or football goalpost) can be found intwo images captured by a camera from two different viewpoints as it ismoving, then it is possible to determine the depth to the object in thetwo images using triangulation or other similar techniques assuming thatthe velocity of the camera is known. In one embodiment, the processingsystem 46 is configured to determine its velocity based on flightsensors (e.g., airspeed and heading sensors) or otherwise (e.g., changein coordinates from a location sensor), to identify a stationary objectin multiple images, and to use such information to calculate depth forthe pixels of the images captured by the drone 15.

Using cameras on multiple athletes at a sporting event, as describedabove, may provide additional data to facilitate understanding ofcomplex situations or provide more content for entertainment, training,or other purposes. As an example, in a football game, cameras may beattached to a quarterback and the receivers that are running routes. Thecameras on the receivers could be used to capture images of defendersattempting to guard or “cover” the receivers. Based on the positions ofthe defenders relative to each receiver as determined from the imagescaptured by the receivers' cameras and/or the quarterback's cameras, theprocessing system 46 may select one or more optimal receivers forreceiving a pass from the quarterback.

Note that the receiver selection may be based on other factors, such asattributes about the defenders collected by the processing system 46during gameplay or otherwise determined by the processing system 46. Asan example, the processing system 46 may maintain data about eachdefender indicating various performance attributes about the defender,such as his peak speed, jumping height, reach, or other parameterindicative of the defender's ability, such as a subjective or objectiverating, referred to hereafter as defender rating,” indicative of thedefender's effectiveness at covering receivers. Such data may bepredefined (e.g., stored in the processing system 46 prior to gameplayor may be determined by the processing system 46 by monitoring thedefenders during gameplay). Using such data, the processing system 46may analyze the abilities of the defender and his relative location to areceiver to determine a value indicative of a probability that a passthrown to the receiver will be completed.

For example, based on the separation distance between the receiver andthe defender and a maximum vertical reach and/or jumping height of thedefender, the processing system 46 may determine whether the defender iscapable of blocking a pass thrown along a trajectory to reach thereceiver as he is running down the field taking into account the factthat the receiver and the defender will likely continue to run duringthe pass until the ball reaches the receiver and/or defender. Based onthe difference between the likely trajectory of the football and thevertical reach of the defender at a point where the football is likelyto reach the defender, the processing system 46 may calculate a value,referred to hereafter as “completion indicator,” indicating theprobability that the pass will be completed. In this regard, a greaterdistance between the trajectory and the defender's vertical reachgenerally gives the quarterback a greater margin of error in attemptingto complete the pass thereby increasing the probability that the passwill be successful.

The completion indicator may also be based on other factors. As anexample, the completion indicator may be lowered for a defender that israted higher than other defenders, as indicated by their respectivedefender ratings. The completion indicator may also be controlled basedon how the defender previously performed in similar situations. As anexample, during gameplay, the processing system 46 may track and storeattribute data of a defender for each pass that he defends. Such datamay include the separation distance between the defender and receiver atthe time a pass is thrown and/or other data, such as whether the passwas completed, whether the defender is guarding the receiver's insideshoulder or outside shoulder, and the maximum speed difference betweenthe defender and the receiver. The processing system 46 analyzes theattribute data to find situations similar to current set of playattributes and analyzes how the defender performed during the similarsituations (such as the completion percentage for the identifiedsituations). Based on such analysis, the processing system 46 may adjustthe completion indicator accordingly. The completion indicator may bebased on many other factors in other embodiments.

Note that the completion indicator may also be based on attributesassociated with the receiver, such as the receiver's speed, jumpingheight, vertical reach, etc. Like the defender, the processing system 46may maintain attribute data on the receiver and search such attributedata to determine how the receiver performed in similar situations inthe past and adjust the completion indicator accordingly. In analyzingthe attribute data, the outcome of similar situations involving the samedefender and receiver may be weighted higher than outcomes of othersituations.

The completion indicator is essentially an assessment by the processingsystem 46 of how likely it is that a pass directed to the receiver willbe completed based on the relative locations of the defender andreceiver and/or other factors, such as the abilities of the defender andreceiver and the performance record of the defender and receiver insimilar situations. Such completion indicator may be used by theprocessing system 46 to select an optimal receiver for catching a pass.As an example, the receiver associated with the highest completionindicator may be selected as the optimal receiver for catching a pass.In other embodiments, other techniques for selecting the optimalreceiver or set of receivers for receiving a pass from the quarterbackare possible.

After selecting one or more optimal receivers, the processing system 46may transmit to a mobile device on the quarterback data indicative ofthe selected receivers to help the quarterback to select a receiver forreceiving a pass during gameplay. As an example, headgear (a displaydevice integrated with his helmet or eyeglasses worn by the quarterback)may display a graphic for identifying the receiver or receivers selectedby the processing system 46 in an augmented reality environment. In thisregard, the display device may project an image into the quarterback'seyes or on eyeglasses worn by the quarterback such that a graphicalelement appears to be superimposed on the selected receiver or receiversthereby indicating which receivers have been determined to be optimalfor receiving a pass. In another example, an image may be projected suchthat a selected receiver or a portion of the football fieldcorresponding to (e.g., at or close to) a selected receiver may appearto be highlighted or colored in a certain manner different than otherportions of the football field. In other embodiments, other techniquesfor identifying the selected receivers are possible. Using theinformation from the processing system 46, the quarterback may selectone of the receivers for the current play and throw the football to theselected receiver.

In addition to assisting the quarterback to select a receiver, theprocessing system 46 may assist the quarterback is selecting a suitabletrajectory. In this regard, the processing system 46 may detect thelocation of defenders and, based on such locations, identify at leastone trajectory for successfully completing a pass while avoidingdefenders. Similar to the putting example described above, theidentified trajectory or trajectories may be displayed to thequarterback in an augmented-reality environment where virtual curvesshow the identified trajectories. The quarterback may select one of thedisplayed trajectories and attempt to throw the football such that itfollows the selected trajectory. As indicated above for putting, morethan one trajectory may be displayed, and the trajectories may be colorcoded or otherwise marked based on the strength of the pass and/or therelease angle, which both impact the trajectory of a pass.

In one embodiment, the trajectory or trajectories displayed to thequarterback are selected based on his performance assessed by the systemin throwing previous passes. In this regard, by monitoring thequarterback's performance over time, the processing system 46 can learnperformance limitations associated with the quarterback, such as thequarterback's arm strength (e.g., how fast or far the quarterback iscapable of throwing a football). Based on such limitations, theprocessing system 46 may eliminate at least some trajectories that aredeemed not to be feasible for the quarterback's capabilities. As anexample, a trajectory that may require a loft (trajectory height) anddistance exceeding the capabilities of the quarterback may be omittedfrom the trajectories displayed to the quarterback. Thus, the processingsystem 46 only displays to the quarterback trajectories that thequarterback is deemed capable of making based on prior throwingperformance. Accordingly, the quarterback is more likely to select atrajectory that will result in a successful outcome (e.g., passcompletion). In addition, based on several factors, such as the possibletrajectories calculated for the pass and the quarterback's pastperformance, the processing system 46 may select a trajectory that isdeemed to be optimal (e.g., result in the highest probability for asuccessful outcome). Such optimal trajectory may be color coded adifferent color than other displayed trajectories or otherwisehighlighted by the processing system 46 such that the quarterback caneasily discern which trajectory is deemed to be optimal.

Note that similar techniques may be used for other types of players. Asan example, the possible trajectories of a field goal attempt may bedisplayed to a kicker. In addition to selecting or otherwise definingthe possible trajectories based on the kicker's prior kick performance,as monitored by the system, the processing system 46 may also receiveinputs indicative of meteorological conditions, such as wind speed, andcompensate the trajectories for wind speed. As an example, a sensormounted on the goalposts or otherwise positioned close to the goalpostsmay measure wind velocity and wirelessly transmit to the processingsystem 46 data indicative of the measured wind velocity, which may beused by the processing system 46 to calculate at least one kicktrajectory. Thus, a kicker may see at least one trajectory, as adjustedfor wind, in order to successfully kick the football through thegoalposts. Such information may be useful for helping the kicker toadjust his kick in order to compensate for wind conditions. Further, thekicker may be informed that his longest possible trajectory is likelyunable to reach the goalposts based on the distance to the goalpost, thecurrent wind conditions, and the kicker's past kick performance. Suchinformation may be useful for affecting certain game-time decisions,such as whether to attempt a field goal during gameplay.

In other embodiments, other techniques for assisting in decisions byathletes during gameplay are possible. As an example, in basketball,similar techniques may be used to analyze the locations of defendersrelative to the locations of teammates, as well as the abilities ofdefenders and/or teammates or the performance of defenders and/orteammates in similar situations, to select and identify which teammateis optimal for receiving a pass. Note that the teammates' locationsrelative to the basketball goal may be used as a factor in selecting ateammate. For example, if multiple teammates are determined to be openfor a pass, the teammate that is closest to the basketball goal may beselected for receiving the pass. Alternatively, if there is an unguardedteammate at the three point line, such teammate may be selected forreceiving the pass even if there is an unguarded teammate closer to thebasketball goal. The selection of a teammate for receiving a pass may bebased on other factors, such as the teammate's past performance. As anexample, by tracking the players over time, the processing system 46 maydetermine the shooting percentage of each player from different areas onthe court. The teammate selected for receiving the pass may be based onsuch data. As an example, the processing system 46 may select theteammate that is (1) capable of successfully receiving a pass (asdetermined based on his location relative to the locations of defenders)and (2) is associated with the highest shooting percentage from hiscurrent location relative to the shooting percentages of the otherteammates that are also capable of successfully receiving a pass.

It should be noted that the monitoring techniques described herein maybe applied to participants of eSports, which generally refers to videogame competitions pertaining to a sport. In an eSport event,participants typically play against each other in a video game featuringa particular sport, such as baseball, football, basketball, wrestling,street fighting, etc., while spectators watch the participants andgameplay. Like traditional sports, large numbers of spectators oftenattend or watch an eSport event. At least one camera may be positionedto capture a facial image of a participant of an eSport event. Suchcamera may be positioned on a drone 15, as described above, but otherlocations of the camera are possible in other embodiments. Such facialimage may be analyzed to estimate the state of mind of the participant.In one embodiment, the facial images captured by the camera may bedisplayed within a video feed of the eSport event. As an example, thefacial images may be shown to spectators at the event or located at aremote location from the event.

In addition, the video images may be analyzed by the processing system46 to determine screen analytics, eye movement, and facial expression inan effort to ascertain the characteristics of good eSport players,including muscle memory, focus, and reaction time. Note that the videoimages may be displayed to the players, as may be desired. As anexample, the face of one player (a “competitor”) may be displayed toanother player so that the other player may get a sense of the mentalstate of his competitor during gameplay. For example, video images ofthe competitor may be superimposed or otherwise combined with the videogame images rendered to the player. Specifically, the video images ofthe competitor may be displayed within a window on the same displayscreen that is used to display the video game to the player.Alternatively, the video images of the competitor may be separatelydisplayed so that it is viewable by the player during gameplay. As anexample, the competitor may be displayed on a separate display screen orwithin an augmented-reality environment where the video game isdisplayed by a physical display unit (such as a desktop monitor ortelevision) while the video images of the competitor are displayed byheadgear (e.g., eyeglasses) worn by the player. Yet other techniques fordisplaying video images of the competitor are possible in otherembodiments. Also, in addition to or in lieu of displaying video imagesof the competitor, data collected about the competitor may be displayedto a player according to the same or similar techniques.

FIG. 11 depicts an exemplary system 300 for monitoring objects inathletic playing spaces, such as football fields, soccer fields,basketball courts, etc. For illustrative purposes, the system 300 willbe described in detail in the context of monitoring basketball playersor basketballs as the players or basketballs are moving within theperimeter of a basketball court. However, the system 300 may be used forother sports, such as football, baseball, hockey, soccer, volleyball,tennis, golf, or any other sport or event in which it is desirable totrack moving objects.

As shown by FIG. 11, the system 300 comprises a sensing system 312 thatis communicatively coupled to a processing system 46. The sensing system312 is configured to sense an object, such as a basketball player orbasketball, that is moving within the athletic playing space, and toprovide sensor data 49 indicative of the locations of the object as itmoves. If desired, the sensing system 312 may reside on a drone 15, asdescribed above, but it is possible for the sensing system 312 to be atother locations in other embodiments. As an example, the sensing system312 may be implemented at a fixed location in the vicinity of a playingspace, or the sensor sensing system 312 may be wearable such that it maybe worn by a player participating in a sporting event. Yet otherlocations and configurations of the sensing system 312 are possible inother embodiments.

As described above, the processing system 46 is configured to receivethe sensor data 49 and analyze the data 49 to determine performanceparameters indicative of the performance of a player. As an example, thesensing system 312 may sense the locations of the player or a portion ofa player's body, and the processing system 46 may analyze the sensordata to determine a velocity, acceleration, or displacement of theplayer or a portion of the player's body (such as a hand or elbow duringa basketball shot). Various performance parameters and techniques formonitoring objects in athletic playing spaces are described by: U.S.Pat. No. 8,622,832, entitled “Trajectory Detection and Feedback System”and issued on Jan. 7, 2014, which is incorporated herein by reference;U.S. Pat. No. 8,617,008, entitled “Training Devices for Trajectory-BasedSports” and issued on Dec. 31, 2013, which is incorporated herein byreference; U.S. patent application Ser. No. 12/127,744, entitled“Stereoscopic Image Capture with Performance Outcome Prediction inSporting Environments” and filed on May 27, 2008, which is incorporatedherein by reference; and U.S. Pat. No. 8,948,457, entitled “True SpaceTracking of Axisymmetric Object Flight Using Diameter Measurement” andissued on Feb. 3, 2015, which is incorporated herein by reference.

In one example, the processing system 46 identifies an object in freeflight, such as a basketball that is traveling toward a hoop of abasketball goal during a basketball shot, and determines the object'slocations in 3D space for a series of image frames. Each such determinedlocation shall be referred to herein as a “measured trajectory point.”Based on the measured trajectory points, the processing system 46determines a trajectory curve representing a path of movement of theobject for the purpose of calculating one or more performanceparameters. As an example, based on the determined trajectory curve, theprocessing system 46 may estimate the object's angle of entry into ahoop of the basketball goal by determining the curve's angle relative toa horizontal plane defined by the hoop at a point that is close to thehoop (e.g., within a plane of the hoop). Note that the processing system46 has a finite number of measured trajectory points depending onvarious factors, such as the frame rate of the camera 351 and the amountof time that the object is in view of the camera 351, and the processingsystem 46 may perform a curve fit algorithm or other type of algorithmin the trajectory analysis in order to smooth the trajectory curve. Thealgorithm for estimating the trajectory of the object can be greatlysimplified if the direction of gravity is known. Indeed, if thedirection of gravity is known, the processing burden for estimating thetrajectory curve can be reduced, and a more accurate trajectory curvecan be determined with fewer measured trajectory points.

As shown by FIG. 11, the processing system 46 is communicatively coupledto an output device 317, such as a display device or an audio device(e.g., a speaker), that is controlled by the processing system 46 toprovide feedback to the player indicative of the player's performanceduring a basketball shot or other activity. As an example, theprocessing system 46 may determine a performance parameter associatedwith a basketball shot, such as release height, release angle, velocity,acceleration, maximum shot height, location of shooter (e.g., horizontaldistance of shooter from hoop when making a shot), make/miss status, orentry angle or velocity of the basketball into a hoop of a basketballgoal. Such performance parameter may be communicated to the player viathe output device 317. If desired, the processing system 46 maycommunicate with the output device 317 through a network (not shown inFIG. 11), such as the Internet or a LAN.

In one exemplary embodiment, the processing system 46 uses the make/missstatus or other information in order to determine various statisticsthat may be useful for characterizing the shooter's skill level over aplurality of shots. As an example, the processing system 46 may countthe total number of shots that a particular shooter takes and also countthe total number of shots made. The processing system 46 may thencalculate a performance parameter based on both counts. As an example,the processing system 46 may calculate the percentage of shots made bydividing the total number of made shots to the total number of shotstaken.

Note that a user sometimes makes a shot without the ball entering thehoop directly. As an example, a ball may strike the hoop and bounceupward before ultimately falling through the hoop for a made basket.Such shots that bounce off of the hoop in an upward direction butultimately pass through the hoop shall be referred to herein as“non-guaranteed makes.” For a non-guaranteed make, it is possible forthe basketball to bounce off of the hoop several times before ultimatelypassing through the hoop. For other shots, sometimes referred to as“swish” shots, the basketball may pass through the hoop without touchingthe hoop. For yet other shoots, the basketball may contact the hoop asit passes downward through the hoop without bouncing off the hoop in anupward direction. Shots for which the basketball passes through the hoopwithout bouncing off the hoop in the upward direction shall be referredto herein as “guaranteed makes.” Note that guaranteed makes includeswish shots for which the basketball does not contact the hoop, as wellas shots for which the basketball contacts the hoop on its way downthrough the hoop without bouncing off the hoop in an upward direction(i.e., away from the floor of the court).

It is believed that the number of guaranteed makes may be a betterindicator of skill level than the number of total made shots. In thisregard, a player that has a higher percentage of guaranteed makes tendsto be a more consistent and better shooter. Moreover, during any givensample period, a lower skill player may appear to be a better shooterthan his or her actual skill level due to an inordinate number ofnon-guaranteed makes, which have less predictable outcomes relative toguaranteed makes. Moreover, the total number of guaranteed makes or aparameter based on the total number of guaranteed makes may constituteone or more of the performance parameters calculated by the processingsystem 46. As an example, the processing system 46 may calculate thepercentage of guaranteed makes by dividing the total number ofguaranteed makes counted during a sample period by the total number ofshots attempted by the same player during the sample period. In otherembodiments, other parameters based on the number of guaranteed makescounted by the processing system 46 are possible.

Note that a performance parameter based on the number or percentage ofguaranteed makes may be reported as feedback to a user. In oneembodiment, a performance parameter based on the guaranteed makescounted by the processing system 46 is used to determine a skill levelfor the player. In this regard, as part of the feedback, the processingsystem 46 may provide a skill level assessment for a particular player.Such skill level assessment may be qualitative or quantitative innature. As an example, the assessment may have various qualitativelevels, such as “poor,” “good,” “great,” and “expert,” and theprocessing system 46 may use the total number of guaranteed makes duringa sample period at least as a factor in selecting which level isappropriate for the player. In this regard, a higher percentage ofguaranteed makes generally results in the selection of a higher-skilllevel according to a predefined algorithm for selecting skill level. Theskill level assessment may also be quantitative in nature, such as ascore from 0 to 100 (or some other range). Generally, the player isassigned a higher score when he or she has a higher percentage ofguaranteed makes, noting that the score may also be based on otherfactors. In any event, the processing system 46 distinguishes betweenguaranteed makes and non-guaranteed makes and ultimately assigns theplayer a skill level assessment based at least on the number ofguaranteed makes counted for the player during a sample period.

If desired, the processing system 46 may store data indicative of theperformance parameters in memory 125 or transmit such data to anotherdevice for storage or analysis. Such data may be analyzed at a latertime for providing feedback, as described herein, or for other purposessuch as for providing information on the play of the game. As anexample, the position of the ball may be compared to the position of anobject associated with the playing space, such as a goal or a boundary,to determine whether the ball crossed or reached the object. Variousother uses of the data processed by the processing system 46 arepossible in other embodiments.

In one example for which the system 300 is used in basketball, theprocessing system 46 is configured to identify a three-point line in thecaptured images. As known in the art, a three-point line is generally anarc that extends from the baseline of a basketball court to the top ofthe key and back to the baseline. The processing system 46 alsoidentifies a shooter that is shooting a basketball near the three-pointline. For example, by tracking the relative locations of athletes to thebasketball, the processing system 46 can determine when one of theathletes shoots the basketball toward a hoop. The processing system 46is configured to identify the feet of such shooter and to determinewhether both of his feet are on a side of the three-point line in athree-point zone (i.e., a zone of the basketball court outside of thearea between the three-point line and the baseline) where shots areworth three points. Based on the relative locations of the shooter'sfeet and the three-point line, the processing system 46 determines aperformance parameter indicating whether the shot is a three-point shot.If any portion of his foot is on or inside the three-point line, theprocessing system 46 determines that the shooter is not taking athree-point shot. Otherwise, the processing system 46 determines thatthe shooter is taking a three-point shot. In such an embodiment, areferee or other user may utilize feedback indicative of the performanceparameter to determine whether to award three points for the basketballshot.

In the context of football, the location of the football may be comparedto a boundary line, such as the goal line, in order to determine whetherany portion of the football reaches or crosses the goal line. That is,based on the images captured by the sensing system 312, the processingsystem 46 may automatically determine whether a touchdown is scored. Insuch an embodiment, a referee or other user may utilize feedback fromthe system 300 in order to determine whether to award points for thefootball reaching or crossing the goal line. In other embodiments, yetother decisions may be made based on comparisons of objects to markingson the playing surface 382.

Note that the processing system 46 may be coupled to the sensing system312 and/or the output device 317 via physical connections (e.g., wires)or wirelessly. In one exemplary embodiment, the sensing system 312 ismounted on a basketball goal, as will be described in more detailhereafter with reference to FIG. 3, and wirelessly transmits sensor datato the processing system 46, and the processing system 46 may comprise acomputer system, such as a desktop, laptop, or handheld computer, whichmay be integral with the output device 317. As an example, a softwareapplication on a smartphone or laptop may implement the functionalitydescribed herein for the processing system 46, which may be implementedin hardware or any combination of hardware, software, and firmware. Asmartphone may have a touch-sensitive display or speaker that is used toimplement the output device 317 for providing visual output to theplayer or other user. In other embodiments, it is unnecessary for theprocessing system 46 to be integral with the output device 317. As anexample, the output device 317 may be implemented via a display screenor an audio device of a smartphone, and the processing system 46 maywirelessly transmit feedback information to the smartphone, whichrenders the feedback information to a user via the output device 317. Inanother embodiment, the output device 317 may be a peripheral deviceconnected to the processing system 46. Yet other configurations arepossible in other embodiments.

FIG. 12 depicts the processing system 46 for an embodiment in which theprocessing system 46 process information for tracking the performance ofone or more players at an athletic playing space and determining adirection of gravity within the athletic playing space, as will bedescribed in more detail below. In the exemplary embodiment shown byFIG. 12, the sensor data 49 comprises image data 349 from a camera (notshown in FIG. 12) and a depth map 350 from a depth sensor (not shown inFIG. 12), but other types of sensor data 49 may be used in otherembodiments.

If desired, the sensing system 312 (FIG. 11) may include any sensor forassisting with the operation and algorithms of the processing system 46.As an example, an accelerometer or other type of motion sensor may beused to provide input regarding movement of the sensing system 312 or acomponent of the sensing system 312, such as the camera 351. Inaddition, one or more orientation sensors, such as tilt sensors orgyroscopes, may be used to provide information about the orientation ofthe sensing system 312 or a component of the sensing system 312, such asthe camera 351. Known algorithms may be used by the control logic 122 inorder to determine the direction of gravity based on accelerometerreadings or other types of readings from motion sensors, orientationsensors, or other types of sensors. As will be described in more detailbelow, the control logic 122 may determine the direction of gravitybased on one or more accelerometers or other types of sensors and usethis information to assist with its operation.

Various types of sensing systems 312 may be used to sense the objectbeing monitored. In one exemplary embodiment, as shown by FIG. 11, thesensing system 312 comprises a camera 351 and a depth sensor 47. Thecamera 351 is configured to capture video images of the playing spacingincluding images of the object being monitored and to provide image data349 defining frames of the captured images. In one embodiment, theimages are two dimensional, and the depth sensor 47 is used to sensedepth or in other words a distance from the sensor 47 to an object inthe image. In this regard, for each frame of image data 349, the depthsensor 47 provides a depth map indicating a respective depth for eachpixel of the image frame. Note that the depth sensor 47 may be orientedsuch that the distance measured by the depth sensor 47 is in a directionthat is substantially normal to the plane of the 2D coordinate systemused by the camera 351, although other orientations of the depth sensor47 are possible in other embodiments.

Various types of cameras 351 and depth sensors 47 may be used toimplement the sensing system 312. In one exemplary embodiment, thesensing system 312 is implemented using a KINECT® camera system sold byMicrosoft Corporation. In such a system, the camera 351 and depth sensor47 are integrated into the same housing 355 (FIG. 3). The camera 351 isconfigured to capture a video stream comprising frames of video data inwhich each frame is defined by a plurality of pixels. Each pixel isassociated with two coordinates, an x-coordinate and a y-coordinate,representing a location in 2D space. For each frame, each pixel isassigned a color value (which may include a red component (R) value, ablue component (B) value, and a green component (G) value) indicative ofthe color of light received by the camera 52 from the location in 2Dspace corresponding to the pixel's coordinates. Further, for each pixel,the depth sensor 47 measures the distance from the sensor 47 to the realworld object that is at the pixel's corresponding location in 2D space.Such distance (which, as described above, may be in a directionsubstantially normal to the plane of the 2D coordinate system used bythe camera 351) may be referred to as the “depth” of the correspondingpixel. Using the image data from the camera 351 and the depth data fromthe depth sensor 47, the location of an object captured by the camera351 can be determined in 3D space. That is, for a point on the object,its x-coordinate and y-coordinate from the image data provided by thecamera 351 indicate its location along two axes (e.g., the x-axis andy-axis), and the point's depth value from the depth sensor, which may bereferred to as the “z-coordinate,” indicates its location along a thirdaxis (e.g., the z-axis). Notably, the coordinate system defined by thethree axes is not relative to gravity. That is, depending on theorientation of the system 312, gravity may be in any direction relativeto the axes of the coordinate system. Thus, unless a calibration processis performed, the direction of gravity relative to the coordinate systemis unknown.

In a Kinect® camera system, the depth sensor 47 comprises a wave emitter363 (e.g., an infrared laser projector or other type of emitter) and asensor 364 for sensing reflections of the energy emitted by the emitter363. The emitter 363 emits infrared radiation at various wavelengthsinto free space, although radiation at other wavelengths outside of theinfrared spectrum (e.g., visible light) may be emitted in otherembodiments, and the sensor 364 senses the reflected energy to capture avideo stream comprising frames of video data. Each frame of the depthdata from the sensor 47 corresponds to a respective frame of image datafrom the camera 351. Further, a pixel of a frame of the depth datacorresponds to (e.g., has the same x- and y-coordinates) and indicatesthe depth for at least one corresponding pixel in the image data fromcamera 351.

In this regard, for a frame of video data captured by the depth sensor47, the depth sensor 47 converts the frame to a depth map 350 byassigning each pixel a new color value (referred to herein as “depthvalue”) representative of the pixel's depth. Thus, when the depth map350 is displayed, objects displayed as the same color within the imageshould be approximately the same distance away from the depth sensor 47,noting that it is often unnecessary for the depth map 350 to actually bedisplayed during operation.

As described above, a given pixel of the image data 349 from the camera351 is associated with an x-coordinate and y-coordinate indicative ofthe pixel's location in 2D space, and the pixel is associated with adepth value from a corresponding pixel in the depth map 350 provided bythe depth sensor 47 indicative of the pixel's z-coordinate. Thecombination of the x-coordinate, y-coordinate, and z-coordinate definesthe pixel's location in 3D space relative to the coordinate system ofthe camera 351. That is, the x-coordinate, y-coordinate, andz-coordinate define the location of the point from which light measuredfor the pixel was reflected toward the camera 351 from an object.

The fact that the direction of gravity is unknown in the coordinatesystem of the camera 351 is not a drawback in many applications for thesensing system 312. However, when the sensing system 312 is used toestimate the trajectory of an object in free flight, as describedherein, knowledge of the direction of gravity relative to the object'sposition is desirable in order to facilitate the process of estimatingthe object's trajectory.

In one exemplary embodiment, the control logic 122 is configured toautomatically determine the direction of gravity relative to thelocations indicated by the sensor data 49 in order to convert the data'scoordinate system into a gravity-based coordinate system. As usedherein, a “gravity-based” coordinate system is a coordinate system forwhich there is known relationship between the direction of gravity andthe axes of the coordinate system such that the direction of gravityrelative to any point indicated by the coordinate system can bedetermined. As an example, a gravity-based coordinate system may bedefined such that the direction of gravity is parallel to an axis (e.g.,z-axis) of the coordinate system, though it is possible for otherrelationships to exist between the direction of gravity and the axes ofthe coordinate system.

Exemplary techniques for converting the sensor data 49 (e.g., image data349 and depth maps 350) from a format relative to the coordinate systemof the camera 351 into a format relative to a gravity-based coordinatesystem will be described in more detail below. In one embodiment, thesensing system 312 is positioned such the camera 351 and depth sensor 47have a broad view of the athletic playing space, including the playingsurface (e.g., surface of a field or court) on which the athleticactivity is played. For example, in basketball, the sensing system 312may be mounted such that the camera 351 and depth sensor 47 arepositioned above the hoop of a basketball goal with a view of the hoopand the floor of the basketball court. FIG. 13 depicts an exemplaryembodiment in which the sensing system 312 is mounted above a hoop 371and backboard 373 of a basketball goal 377. As an example, the goal 377,including the backboard 373 and hoop 371, may be mounted on one or morepoles 379 that extend from a ceiling or wall of a building or otherstructure, and the sensing system 312, including the camera 351 anddepth sensor 47, may be mounted on at least one such pole 379 above thebackboard 373. As shown by FIG. 13 the hoop is coupled to the backboard373 by a bracket 383, and a net 384 may be coupled to and hang from thehoop 371.

Further, the sensing system 312 may be oriented such that the camera 351and depth sensor 47 have a downward view that includes the hoop 371, aswell as at least a portion of the playing surface 382 (which is thefloor surface of the basketball court in the current example). When thesensing system 312 is so oriented, the camera 351 and depth sensor 47capture images of the playing surface 382 and other objects, such asgoal 377, within the athletic playing space, as shown by block 502 ofFIG. 17.

FIG. 14 shows an exemplary depth map image that may be captured by thedepth sensor 47 in such an embodiment. In the depth map image shown byFIG. 14, the pixels are colored based on depth, as determined by thedepth sensor 47. In this regard, the darker that a pixel's color is inthe depth map 350, the smaller is the pixel's depth value. Thus, pixelscorresponding to objects closer to the depth sensor 47 appear darker incolor relative to pixels corresponding to object farther from the depthsensor 47. As an example, because the hoop 371 and backboard 373 arecloser to the depth sensor 47 relative to the playing surface 382,pixels defining an image of the hoop 371 and backboard 373 are coloreddarker than pixels defining an image of the playing surface 382.

In one exemplary embodiment, the control logic 122 analyzes the depthmap 350 in order to identify a playing surface (PS) plane within theimage of the depth map 350, as shown by block 505 of FIG. 17. The PSplane generally refers to a plane that is parallel to the playingsurface 382 (e.g., surface of a court or field) on which the athleticactivity is played. In this regard, athletic activities are often playedin wide, open spaces with relatively flat surfaces, such as fields orcourts. Thus, a large number of pixels in the depth map shouldcorrespond to the playing surface 382 and, hence, be within the sameplane. For example, when the sensing system 312 is mounted high abovethe playing surface, a significant portion of an image may correspond tothe playing surface, and the pixels corresponding of the playing surfacemay have color values within a relatively narrow color range. Moreover,the control logic 122 is configured to analyze the depth map 350 toidentify planes. That is, the control logic 122 is configured toidentify at least one set of depth pixels that are within the sameplane. When the sensing system 312 is mounted high above the playingsurface, planes can be identified by finding groups of contiguous depthpixels having similar color values. However, other techniques may beused in other embodiments. As an example, the surface geometry of anobject within view of the camera 351 can be analyzed based on the depthpixels in order to identify depth pixels that are within the same plane.Thus, it is unnecessary for the pixels in the same plane to have thesimilar depths in order to be in the same plane.

As an example, in a volleyball game, one or more sensing systems 312 maybe mounted on one or more sides of the volleyball court such that asensing system 312 is below the net of the volleyball court. In such anembodiment, the view of the floor of the volleyball court may be closerto a horizontal perspective than a vertical perspective such that depthpixels corresponding to the floor of the volleyball court may havesignificantly different depth values as the floor extends away from thesensing system 312.

Some objects, such as portions of the goal 377, may have flat surfacesfrom the perspective of the depth sensor 47, but the size of a flatsurface of the goal 377 within the view of the depth sensor 47 is likelyto be much smaller than the size of the playing surface 382. For eachset of depth pixels defining a plane, the control logic 122 maydetermine the total number of depth pixels within the plane and comparethis number to a threshold. If the number is below the threshold, thecontrol logic 122 may determine that the pixel set does not correspondto the playing surface 382. That is, the size of the plane representedby the pixel set is too small to be representative of the playingsurface 382. The pixel set having the greatest number of depth pixelsabove the threshold within the same plane may be identified by thecontrol logic 122 as the pixel set corresponding to the playing surface382, referred to hereafter as the “floor planes (FP) pixel set.”

Note that various sensors may be used to assist with identification ofthe FP pixel set defining the PS plane. As an example, as describedabove, one or more accelerometers or other types of sensors may be usedto determine the approximate direction of gravity, and such informationmay be used to filter the planes identified by the control logic 122 inorder to eliminate planes that are not within a predefined range of thedirection of gravity, as determined by the foregoing sensors. As anexample, only pixel sets defining a plane that is substantiallyperpendicular to the direction of gravity, as determined by one or moreaccelerometers or other sensors, may be eligible for selection as the FPpixel set. Once the FP pixel set is identified, it may be used to make amore precise measurement of the direction of gravity according to thetechniques described herein.

Due to errors in estimating the pixel depths by the depth sensor 47 orother factors (such as curvature, if any, of the playing surface), theFP pixel set in some cases may not define a perfect plane. The controllogic 122 is configured to perform a mathematical smoothing operation onthe FP pixel set in order to remove outliers far away from the FP pixelset, as shown by block 508 of FIG. 17. In one exemplary embodiment,Random Sample Concensus is used to implement the mathematical smoothingoperation, but other types of smoothing operations may be used in otherembodiments.

In addition to the smoothing operation, the control logic 122 alsoperforms an algorithm, referred to herein as “floor differencing,” in aneffort to remove depth pixels that are out of the PS plane but closer tothe PS plane than the outliers removed by the smoothing operation, asshown by block 511 of FIG. 17. In this regard, after performing thesmoothing operation, the control logic 122 analyzes the FP pixel set inorder to estimate the initial location and orientation of the PS plane,which will be referred to as the “initial PS plane.” The control logic122 then compares each depth pixel of the FP pixel set to the initial PSplane identified by the control logic 122. As an example, the controllogic 122 may determine the difference between (1) the depth indicatedby the depth pixel and (2) the depth of the point on the initial PSplane that is closest to the depth indicated by the depth pixel. If thedifference is greater than a predefined threshold (TH), then the controllogic 122 removes the depth pixel from the FP pixel set. Thus, byperforming the floor differencing, depth pixels that are associated withlocations greater than a threshold distance away from the initial PSplane are removed from the FP pixel set.

After performing floor differencing, the control logic 122 againanalyzes the FP pixel set in order to estimate the location andorientation of the PS plane indicated by the modified FP pixel set,thereby identifying the PS plane that is to be used for converting thesensor data 49 into a format relative to the gravity-based coordinatesystem. In this regard, the control logic 122 may determine that thedirection of gravity is perpendicular to this identified PS plane, asshown by block 514 of FIG. 17.

Before the sensor data 49 is converted, the control logic 122 isconfigured to select an origin for the gravity-based coordinate systemand to define three axes: an x-axis, a y-axis, and a z-axis. These axesare perpendicular to each other, and each axis is defined to passthrough the origin. In one embodiment, the x-axis and y-axis are definedto be parallel to the identified PS plane, and the z-axis is defined tobe perpendicular to the PS plane and, therefore, parallel with thedirection of gravity. In other embodiments, other orientations of theaxes relative to the direction of gravity and the PS plane are possible.

In order to facilitate calculations of performance parameters, thecontrol logic 122 is configured to define a relationship between thegravity-based coordinate system and the athletic playing environment. Asan example, in order to determine the angle of entry of a basketballinto a hoop 371, the control logic 122 should be aware of the locationsof the basketball relative to the hoop 371 as it is traveling along atrajectory. This may be achieved by determining a relationship betweenat least one reference point in the gravity-based coordinate system,such as the origin, and at least one reference point in the athleticplaying environment. By doing so, the location of any object sensed bysensing system 312, such as a player or basketball, relative to otherobjects in the playing environments, such as a hoop 371, can beautomatically determined.

Note that any point in the playing environment can be used as areference for the gravity-based coordinate system. As an example, withinthe image data 349, it is possible to identify a boundary line or othercourt marking on the floor of a basketball court and use the identifiedmarking to reference the gravity-based coordinate system to the playingenvironment. However, the types or styles of markings may vary fromcourt-to-court. A basketball hoop 371, on the other hand, generally hasa consistent size and shape, thereby facilitating identification of ahoop 371 within the images provided by the sensing system 312.

The control logic 122 is configured to identify a reference object(e.g., a basketball hoop 371) in the images provided by the sensingsystem 312 and to reference the gravity-based coordinate system based onthe identified object, as shown by block 515 of FIG. 17. In oneexemplary embodiment, the control logic 122 is configured to locate abasketball hoop 371 in the images and to define the gravity-basedcoordinate system such that its origin is at the center of such hoop371. Notably, the plane of the basketball hoop 371 should be parallel tothe PS plane identified by the control logic 122. Since the x-axis andy-axis are defined to be parallel to the PS plane, the x-axis and y-axisshould be within the plane of the basketball hoop 371 when the origin ispositioned at the center of the hoop 371. In addition, when the originis so defined, the z-axis passes through the center of the hoop 371 in adirection parallel to gravity.

In order to facilitate installation of the sensing system 312, thesensing system 312 may be mounted at any height above the playingsurface 382 and the hoop 371. Without knowing the distance of the hoop371 from the depth sensor 47, the control logic 122 is configured toanalyze a depth map 350 from the depth sensor 47 in order to estimatesuch distance. Before estimating this distance, the control logic 122first locates the basketball hoop 371 within such image. Exemplarytechniques for identifying the hoop 371 will be described in more detailbelow.

In one exemplary embodiment, the control logic 122 is configured toidentify a pixel set, referred to hereafter as the “basketball goal (BG)pixel set,” that does not include the pixels corresponding to playingsurface 382, thereby removing a significant number of pixels from the BGpixel set. As an example, the control logic 122 may perform an algorithmsimilar to the floor differencing algorithm described above on all ofthe depth pixels of a depth map 350. However, rather than removing thedepth pixels that are greater than a threshold (TH) distance from the PSplane, the control logic 122 instead removes the depth pixels that areless than the threshold distance from the PS plane and keeps the depthpixels that are more than the threshold distance from the PS plane.

FIG. 15 shows an exemplary depth map image after floor differencing hasbeen performed in order to remove the depth pixels that correspond tothe PS plane. As shown by FIG. 15, the depth map image includes an image401 of the hoop 371, an image 402 of a net 384 coupled to the hoop 371,an image 403 of the backboard 373 to which the hoop 371 is mounted, andan image 404 of a bracket 383 for coupling the hoop 371 to the backboard373. From a view above the goal 377, the bracket 383 may appearsubstantially rectangular, as shown by FIG. 15, although other shapesare possible.

The control logic 122 searches the depth map image for the image of acircular ring in order to identify the hoop image 401. When the hoopimage 401 is found, the control logic 122 determines the size (e.g.,diameter) of the hoop image 401. There are various techniques that canbe used to determine the size of the hoop image 401. In one exemplaryembodiment, the control logic 122 superimposes a scalable hoop template411 over the hoop image 401, as shown by FIG. 16 (noting that thetemplate 411 is shown in red in FIG. 16). The diameter of the hoop image411 is adjusted in order to maximize the number of pixels in the hoopimage 401 that are covered by the hoop template 411. Since the actualdiameter of the hoop 371 is known (about 18 inches for a standard-sizedhoop), the distance of the depth sensor 47 from the hoop 371 can becalculated based on the diameter of the template 411.

The hoop diameter in the hoop image 401 may be used by the control logic122 to calibrate the trajectory calculations to account for the locationof the sensing system 312. In this regard, for accurate trajectorycalculations, the control logic 122 should be aware of scaling factor tobe used to relate distances in an image to the physical distances in thereal world. As an example, a distance of half an inch in a capturedimage may represent a distance of several feet (or some other distance)in the real world. The scaling factor between real-world dimensions anddimensions within the captured images is generally based on severalfactors, including the location of the sensing system 312 relative toobjects appearing in the images, as well as the zoom or magnification ofthe camera used to capture the image. In one exemplary embodiment, thecontrol logic 122 determines how distances in the captured imagescorrelate or scale with real world distances based on the hoop image401. In this regard, as described above, the real world diameter of thehoop is typically the same from goal-to-goal (i.e., approximately 18inches). Thus, based on the diameter of the hoop in the image 401, thecontrol logic 122 can determine the appropriate scaling factor forconverting distances in the captured image to real world distances. Inother embodiments, other types of objects having known dimensions may beused instead of the hoop. As an example, certain court markings (such asa length of a free throw line) may be known, and images of such courtmarkings may be used to determine the appropriate scaling factor. Also,the distance from the hoop 371 to the playing surface 382 is usuallyknown and can be used as a reference for determining the scaling factor.In other embodiments, yet other types of objects and dimensions may beused to determine the appropriate scaling factor.

In addition, the control logic 122 is also configured to orient thegravity-based coordinate system based on an image of the sensor data 49,as shown by block 517 of FIG. 17. To achieve this in one embodiment, thecontrol logic 122 is configured to identify the image of the bracket 383in the BG pixel set. As shown by FIG. 15, the region around the hoopimage 401 should be substantially devoid of depth pixels as result ofthe floor differencing described above except for the region where thebracket image 404 is located. Thus, a process for finding the bracketimage 404 should be relatively simple and reliable even for brackets 383of different shapes and configurations. After identifying the bracketimage 404, the control logic 122 is configured to orient the axes of thegravity-based coordinate system based on the position of the bracketimage 404 relative to the hoop image 401. As an example, the controllogic 122 may define one of the axes (e.g., the x-axis) such that itpasses through the center of the hoop 371 and the center of the bracket383.

After orienting the gravity-based coordinate system, the control logic122 is configured to convert the image data 349 and the depth maps 350from a format relative to the coordinate system of the camera 351 to aformat relative to the gravity-based coordinate system, as shown byblock 522 of FIG. 17. Thus, the pixel coordinates for the image data areconverted to be relative to the origin of the gravity-based coordinatesystem rather than the origin of the camera's coordinate system. Itshould be noted that various changes and modification to FIG. 17 wouldbe apparent to a person of ordinary skill upon reading this disclosure.Moreover, any of the steps of FIG. 17 can be omitted and/or the order ofany of the steps may be rearranged as may be desired.

Since the distance of the sensing system 312 from the origin of thegravity-based system is known, the location of any object in the imagedata 349 relative to the hoop 371 or other object in the playingenvironment can be calculated. As an example, the trajectory of abasketball can be compared to the location of the hoop 371 in order todetermine an angle of entry of the basketball into the hoop. In anotherexample, by knowing the location of the hoop 371 relative to the sensingsystem 312, the location of a particular court marking within an image,such as a free throw line, can be determined since the marking of astandard basketball court should be a predefined distance and directionfrom the hoop 371. Thus, it is possible to determine the location of anobject relative to the location of the free throw line. For example, thecontrol logic 122 may determine that a player is shooting a free throwbased on the player's position, as determined from the image data,relative to the free throw line when he or she launches a basketballtoward the hoop 371.

It should be noted that the exemplary processes described above forcalibrating the gravity-based coordinate system and converting sensordata 49 into a format relative to the gravity-based coordinate systemcan be automatically and efficiently performed without any humanintervention and without significant processing burdens relative toother techniques that may exist for calibrating coordinate systems.Thus, the process can be repeated as often as may be desired duringoperation. For example, if the sensing system 312 is struck by abasketball causing the camera 351 and sensor 47 to move, thegravity-based coordinate system can be automatically and quicklyre-calibrated according to the techniques described herein.

In addition, in several examples described above, it is assumed that thedirection of gravity is perpendicular to the PS plane identified by thecontrol logic 122. However, other directions of gravity relative to theidentified PS plane are possible. As an example, certain playingsurfaces may be sloped for various reasons, such as for facilitatingdrainage of water from the surface. For example, a football field oftenhas a “crown” in the middle of the field, and the field slopes downwardaway from the crown as the sidelines are approached. Thus, portions ofthe field close to the sideline may be sloped such that the direction ofgravity is oblique relative to the surface of the field in the slopedregions. In some cases, the slope of the surface may increase closer tothe sideline.

In one exemplary embodiment, the control logic 122 is configured toaccount for the sloped surface when determining the direction ofgravity. Note that there are various techniques that can be used toaccount for the slope of the surface. As an example, the processingsystem 46 may store data, referred to herein as “surface data 252” (FIG.12), indicative of the slope of the playing surface at one or morepoints. As an example, for each of a plurality of locations on theplaying surface, the surface data 252 may have a value indicating adegree to which the surface is sloped, such as a value indicating anangle of the direction of gravity relative to the playing surface atsuch location. Such data may be predefined and stored in memory 125prior to normal operation of the processing system 46. As an example, atleast one image of the playing surface may be captured with a camera 351and the depth sensor 47 during a calibration process, and the images maybe analyzed by the control logic 122 or otherwise to determine the slopeof the playing surface at various locations. In this regard, asdescribed above, depth pixels of depth maps 350 from the depth sensor 47may be correlated with pixels of image data 349 from the camera 351, andthe depths indicated by the depth pixels may be used to calculate theslope of the playing surface at various locations in the images capturedby the camera 351. That is, the playing surface is effectively mapped bythe control logic 122 in a calibration process so that the slope of theplaying surface (relative to gravity) at various locations is indicatedby the data 252. In this calibration process, the direction of gravitymay be determined based on manual input (e.g., a user may provide aninput indicating a direction of gravity within an image) or by findingwithin an image an object of known orientation, as described above forthe basketball hoop. In other embodiments, the sensing system 312 mayhave sensors, such as accelerometers or other types of sensors that canbe used to sense the direction of gravity, as described above.

In one exemplary embodiment, the sensing system 312 is coupled to anaerial vehicle 255, as shown by FIG. 18, in order to perform thecalibration process described above in which the playing surface ismapped to determine its surface topology. As described above, the aerialvehicle 255 may comprise a drone 15 or other type of aircraft that fliesabove the playing surface allowing the camera 351 and depth sensor 47 tocapture images of the playing surface as the vehicle 255 flies. Ifdesired, the vehicle 255 may be coupled to a tether that holds thevehicle 255 in the air and/or guides the vehicle 255 as it moves. Inother embodiments, the vehicle 255 may be untethered so that it canfreely fly under the direction of a pilot or remote control. In suchembodiments, the camera 351 captures images of the playing surface 382from a location above the playing surface 382, and the depth sensor 47measures the depth or distance to the playing surface. Each pixel of theimages captured by the camera 351 is associated with a depth valueindicative of the distance from the surface point represented by thepixel to the sensing system 312. Based on such depth values, the slopeof the surface at various locations can be calculated and stored in thesurface data 252 for later use in determining the direction of gravity,as described above.

During operation after the calibration process described above, thecontrol logic 122 may be configured to determine a location of theplaying surface that is within the image before making a decisionregarding the direction of gravity. As an example, for a given depth map350, the control logic 122 may analyze a corresponding set of image data349 to determine a relative location of the playing surface within theimage defined by this set of image data 349. As an example, based onboundary markings (e.g., sidelines on a football field) within theimage, the control logic 122 may determine that a location of theplaying surface within the image is close to a sideline where theplaying surface is significantly sloped. Based on the surface data 252,the control logic 122 determines the extent of the surface slope at suchlocation and calculates or otherwise determines the direction of gravitybased on the slope. Specifically, the control logic 122 accounts for theslope by assigning, based on the surface data 252, a direction ofgravity that is at an oblique angle relative to the playing surface atthe identified location. Thus, the direction of gravity determined bythe logic 122 should be accurate even though the image used by thecontrol logic 122 is of a sloped region of the playing surface.

Note that the sensing system 312 coupled to the aerial vehicle 255 maybe used in the manner described above in order to monitor athletes onthe playing surface according to the techniques described above. Thealgorithms described above for determining the direction of gravitybased images captured by the camera 351 and the depth sensor 47 may beparticularly useful for such an embodiment. In this regard, as thevehicle 255 is flying, the orientation of the sensing system 312relative to gravity is likely to change frequently and abruptly. Thealgorithms for determining the direction of gravity based on the camera351 and depth sensor 47 may be repetitively and frequently performed(such as multiple times per second) while consuming a relatively lowamount of processing resources yet providing very accurate estimationsof the direction of gravity. Such characteristics may be beneficial in avariety of other applications.

In calculating the trajectory of a moving object, it can generally beassumed that the force exerted on such object by gravity is constant.However, the magnitude of such force generally changes with altitude. Asan example, the magnitude of gravitational pull is slightly differentfor an event occurring in a mountainous region relative to an eventoccurring close to sea level. In one exemplary embodiment, theprocessing system 46 is configured to account for variations in altitudein performing the trajectory calculations.

In this regard, the processing system 46 is configured to storegravitational pull data 352 (FIG. 12) indicative of the magnitude ofgravitational pull for various altitudes. In addition, during operation,the control logic 122 is configured to determine the approximatealtitude of the event being monitored by the processing system 46. As anexample, a user may simply enter the altitude of the event via an inputdevice (not shown), such as a keyboard, keypad, or mouse, of theprocessing system 46, or the processing system 46 may receive suchinformation wirelessly via the wireless communication interface 145.Alternatively, the sensing system 312 may have a sensor (not shown),such as an altimeter or a location sensor (e.g., GPS sensor), that canbe used to automatically determine the approximate altitude of at leasta component of the system 300 and, therefore, the event at which thesystem 300 is located. In other embodiments, other techniques fordetermining altitude are possible.

After determining the altitude, the control logic 122 is configured toconsult the gravitational pull data 352 in order to determine themagnitude of gravitational pull to be used for trajectory calculations.As an example, the data 352 may be implemented as a table of altitudevalues and gravitational pull values, and the control logic 122 may usethe altitude value received from the sensing system 312 or otherwiseobtained by the control logic 122 as a key for looking up theappropriate gravitational pull value to be used for trajectorycalculations. In other embodiments, the control logic 122 may beconfigured to algorithmically calculate the appropriate gravitationalpull value based on the determined altitude. Yet other techniques fordetermining a suitable gravitational pull value for use in trajectorycalculations are possible in other embodiments. By determining thegravitational pull value based the actual altitude of the event beingmonitored, more accurate trajectory calculations are possible, therebyimproving the performance of the system 300.

Various embodiments of monitoring systems are described above in thecontext of basketball. It should be emphasized that similar techniquesmay be used in other sports for defining a gravity-based coordinatesystem and converting sensor data into a format relative to thegravity-based coordinate system. As an example, for football, a sensingsystem 312 may be positioned such that a goalpost and a surface of afootball field are within view. Using techniques similar to thosedescribed above for basketball, a surface plane corresponding to thesurface of the football field may identified and used to determine thedirection of gravity. Further, the shape of the goalpost may be used toorient the gravity-based coordinate system relative to the boundariesand markings of the football field. A goal in hockey may be similarlyused to orient a gravity-based coordinate system. Similar techniques maybe used in yet other sports to define and orient a gravity-basedcoordinate system.

Now, therefore, the following is claimed:
 1. A system for monitoringplayer performance at sporting events, comprising: at least one cameraconfigured to capture images of an object launched by a first playerwhile participating at a sporting event; and at least one processorconfigured to determine a location of the first player when the objectis launched by the first player and to analyze the captured images fordetermining a trajectory of the object launched by the first player, theat least one processor configured to determine a target zone for theobject based on the determined location of the first player and tocompare the trajectory to the target zone, wherein a size of the targetzone is based on the determined location of the first player, the atleast one processor configured to determine a value indicative of aperformance of the first player in launching the object based on thecomparison of the trajectory to the target zone, the at least oneprocessor further configured to provide feedback indicative of theperformance.
 2. The system of claim 1, wherein the at least oneprocessor is configured to determine a distance between the determinedlocation of the first player and the target zone and to determine thesize of the target zone based on the distance.
 3. The system of claim 1,wherein the at least one processor is configured to determine a locationof the target zone based on movement of at least one player at thesporting event.
 4. The system of claim 3, wherein the at least oneplayer includes a receiver for receiving the object from the firstplayer.
 5. The system of claim 4, wherein the at least one processor isconfigured to determine whether the object is received by the receiverbased on the captured images.
 6. The system of claim 4, wherein the atleast one processor is configured to determine the location of thetarget zone based on a velocity of the receiver.
 7. The system of claim4, wherein the at least one processor is configured to determine alocation and a velocity of the receiver and to determine the location ofthe target zone based on the location and the velocity of the receiver.8. The system of claim 7, wherein the at least one processor isconfigured to determine a distance between the determined location ofthe first player and the target zone and to determine the size of thetarget zone based on the distance.
 9. The system of claim 1, wherein theat least one processor is configured to determine whether the trajectoryintersects with the target zone, and wherein the value is based onwhether the trajectory is determined to intersect with the target zone.10. The system of claim 9, wherein the at least one processor isconfigured to determine whether a plurality of trajectories of one ormore objects launched by the first player respectively intersect with aplurality of target zones, and wherein the value represents a percentageof a number of the trajectories that intersect with a respective one ofthe plurality of target zones.
 11. A method for monitoring playerperformance at sporting events, comprising: capturing, with at least onecamera, images of an object launched by a first player whileparticipating at a sporting event; and determining, with at least oneprocessor, a location of the first player when the object is launched bythe first player; analyzing, with the at least one processor, thecaptured images for determining a trajectory of the object launched bythe first player; determining, with the at least one processor, a targetzone for the object based on the determined location of the firstplayer, wherein a size of the target zone is based on the determinedlocation of the first player; comparing, with the at least oneprocessor, the trajectory to the target zone; determining, with the atleast one processor, a value indicative of a performance of the firstplayer in launching the object based on the comparing; and providing,with the at least one processor, feedback indicative of the performance.12. The method of claim 11, further comprising: determining, with the atleast one processor, a distance between the determined location of thefirst player and the target zone; and determining, with the at least oneprocessor, the size of the target zone based on the distance.
 13. Themethod of claim 11, further comprising determining, with the at leastone processor, a location of the target zone based on movement of atleast one player at the sporting event.
 14. The method of claim 13,wherein the at least one player includes a receiver for receiving theobject from the first player.
 15. The method of claim 14, furthercomprising determining, with the at least one processor, whether theobject is received by the receiver based on the captured images.
 16. Themethod of claim 14, wherein the determining the location of the targetzone is based on a velocity of the receiver.
 17. The method of claim 14,further comprising determining, with the at least one processor, alocation and a velocity of the receiver, wherein the determining thelocation of the target zone is based on the location and the velocity ofthe receiver.
 18. The method of claim 17, further comprising:determining, with the at least one processor, a distance between thedetermined location of the first player and the target zone; anddetermining, with the at least one processor, the size of the targetzone based on the distance.
 19. The system of claim 11, wherein the atleast one processor is configured to determine whether the trajectoryintersects with the target zone, and wherein the value is based onwhether the trajectory is determined to intersect with the target zone.20. The method of claim 19, further comprising determining, with the atleast one processor, whether a plurality of trajectories of one or moreobjects launched by the first player respectively intersect with aplurality of target zones, and wherein the value represents a percentageof a number of the trajectories that intersect with a respective one ofthe plurality of target zones.